Recording method, recording device, and program

JPWO2023188652A5Pending Publication Date: 2024-12-09
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024511252
Authority / Receiving Office
JP · JP
Patent Type
Applications
Filing Date
2024-09-12
Publication Date
2024-12-09

AI Technical Summary

Technical Problem

Existing methods for recording supplementary information in image data, such as moving image data, are inefficient in adapting to changes in the subject or scene, leading to suboptimal search and recording of additional information.

Method used

A method and device that recognize subjects in frames, set specific search items based on conditions like image quality and focus position, and record supplementary information, allowing for precise and efficient searching and recording of additional data, with priority settings and adaptive accuracy based on previous search results.

Benefits of technology

Enables efficient and accurate recording of supplementary information, improving the ability to identify and record details about subjects within image data, even as the subject or scene changes, by using a processor to recognize subjects, set search items, and record data based on user-defined conditions.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

Provided are a recording method, a recording device, and program for appropriately recording supplementary information in accordance with a subject in image data. In the present invention, the following steps are executed: a recognition step for recording supplementary information for frames in moving image data consisting of a plurality of said frames, and recognizing a plurality of recognition subjects in the plurality of frames; a search step for searching for the supplementary information recordable for search subjects, which are at least some of the plurality of recognition subjects, on the basis of search items; a setting step for setting a different search item for each search subject if there are a plurality of search subjects; and a recording step for recording at least some of the search items as the supplementary information, on the basis of a result obtained in the search step.
Need to check novelty before this filing date? Find Prior Art

Description

Recording method, recording device, and program

[0001] The present invention relates to a recording method, a recording device, and a program.

[0002] For image data such as moving image data and still image data, additional information about the subject in the data may be recorded. By recording such additional information, it is possible to identify the subject in the image data before using the image data.

[0003] For example, in the invention described in Patent Document 1, at least one keyword is assigned to each scene of a moving image based on a user's operation, and the keyword assigned to each scene is recorded together with the moving image data.

[0004] Japanese Patent Application Publication No. 6-309381

[0005] When adding additional information about a subject in image data, it is necessary to search for information appropriate for the subject (for example, information that matches the characteristics of the subject). In this case, it is necessary to efficiently search for additional information according to the subject. On the other hand, the subject in the image data may change depending on the shooting scene or the orientation of the shooting equipment. In such cases, it is necessary to search for (search for) additional information appropriate for the changed subject.

[0006] One embodiment of the present invention has been made in consideration of the above circumstances, and aims to solve the problems of the prior art described above and provide a recording method, recording device, and program for appropriately recording supplementary information corresponding to the subject in image data.

[0007] In order to achieve the above object, the recording method of the present invention is a recording method for recording supplementary information for frames in video image data composed of a plurality of frames, and is characterized by comprising: a recognition process for recognizing a plurality of recognized subjects within the plurality of frames; a search process for searching, based on search items, for supplementary information that can be recorded for a search subject that is at least a part of the plurality of recognized subjects; a setting process for setting different search items for each search subject when there are a plurality of search subjects; and a recording process for recording at least a part of the search items as supplementary information based on the results of the search process.

[0008] The search process may be performed on a search subject selected according to a predetermined condition. In the above configuration, the condition may be based on image quality information or size information of the search subject in the frame. In the above configuration, the condition may be based on a focus position set in a recording device that records the video data, or a user's gaze position during recording of the video data.

[0009] In the recording step, coordinate information of the focus position or the line of sight position may be recorded as supplementary information for the frame.

[0010] In addition, the search step may use search items selected by the user.

[0011] In addition, in the setting step, a priority may be set for each of the search subjects. In this case, the accuracy of the search items set for the search subjects with higher priorities may be higher than the accuracy of the search items set for the search subjects with lower priorities.

[0012] In addition, in the setting step, the precision of the search items may be set according to the results of a search step that has been executed in the past.

[0013] Furthermore, among the multiple frames, a search subject in a first frame may exist in a second frame that precedes the first frame. In this case, in the setting step, the accuracy of the search item set for the search subject in the first frame may be higher than the accuracy of the search item set for the search subject in the second frame.

[0014] The recording method of the present invention may further include a receiving step of receiving a user's input regarding an item of the additional information. In this case, the recording step may be executed for an input frame corresponding to the user's input among the plurality of frames, and additional information corresponding to the input item may be recorded.

[0015] In addition, in the receiving step, it may be possible to receive additional information items different from the search items set in the setting step.

[0016] The additional information may be stored in a data file separate from the video data.

[0017] A recording device according to one embodiment of the present invention includes a processor and records incidental information for frames in video data consisting of a plurality of frames. The processor executes a recognition process for recognizing a plurality of recognized subjects in the plurality of frames, a search process for searching, based on search items, for recordable incidental information for a search subject that is at least a portion of the recognized subjects, a setting process for setting different search items for each search subject when there are multiple search subjects, and a recording process for recording at least a portion of the search items as incidental information based on the results of the search process.

[0018] Furthermore, a program according to an embodiment of the present invention is a program for causing a computer to execute each of the recognition step, search step, setting step, and recording step included in the above-described recording method.

[0019] A recording method according to one embodiment of the present invention is a recording method for recording ancillary information in image data, and includes a recognition step for recognizing a plurality of recognized subjects in the image data, a search step for searching, based on search items, for ancillary information that can be recorded for a search subject that is at least a part of the plurality of recognized subjects, a setting step for setting different search items for each search subject when there are a plurality of search subjects, and a recording step for recording at least a part of the search items as ancillary information based on the results of the search step.

[0020] 1 is an explanatory diagram of moving image data; FIG. 2 is a diagram showing incidental information related to a subject in a frame; FIG. 3 is a diagram showing an example of hierarchical structured incidental information; FIG. 4 is a diagram showing a procedure for identifying the position of a circular subject area; FIG. 5 is a diagram showing a procedure for recording incidental information for a frame; FIG. 6 is an explanatory diagram of search items; FIG. 7 is a diagram showing a situation in which a subject in a frame is changing while moving image data is being recorded; FIG. 8 is a diagram showing a hardware configuration of a recording device according to an embodiment of the present invention; FIG. 9 is an explanatory diagram of functions of a recording device according to an embodiment of the present invention; FIG. 10 is a diagram showing the relationship between priority for a search subject and search items; FIG. 11 is an explanatory diagram of the accuracy of search items set for a search subject in a first frame and a second frame; FIG. 12 is a diagram showing a situation in which the accuracy of search items is gradually increased; FIG. 13 is a diagram showing the execution rate of a search process when search items selected by a selection unit are used and when search items set by a setting unit are used; FIG. 14 is a diagram showing a recording flow according to an embodiment of the present invention; FIG. 15 is a diagram showing an example in which incidental information is saved in a data file different from the moving image data.

[0021] Specific embodiments of the present invention will be described. However, the embodiments described below are merely examples for facilitating understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved from the embodiments described below without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof.

[0022] In addition, in this specification, the concept of "device" includes not only a single device that performs a specific function, but also a combination of multiple devices that exist independently and in a distributed manner but cooperate (link) to perform a specific function.

[0023] In addition, in this specification, the term "person" refers to an entity that performs a specific action, and this concept includes individuals, groups, corporations such as companies, and organizations, as well as computers and devices that constitute artificial intelligence (AI). AI is a technology that realizes intelligent functions such as inference, prediction, and judgment using hardware and software resources. AI algorithms are arbitrary, and include, for example, expert systems, case-based reasoning (CBR), Bayesian networks, and subsumption architectures.

[0024] <<One Embodiment of the Present Invention>> One embodiment of the present invention relates to a recording method, a recording device, and a program for recording supplementary information to frames in moving image data.

[0025] [Regarding video data and frames] Video data is created by known video capture devices (hereinafter referred to as capture devices) such as video cameras and digital cameras. The capture devices capture images of a subject within a field of view at a constant frame rate (the number of frame images captured per unit time) under preset exposure conditions to generate analog image data (RAW image data). The capture devices then perform correction processes such as gamma correction on the digital image data converted from the analog image data to create frames (more specifically, frame image data).

[0026] Then, the image capturing device records frame image data at a fixed rate (interval), thereby creating moving image data made up of a plurality of frames, as shown in FIG.

[0027] Each frame of video data contains one or more objects, i.e., one or more objects exist within the field of view of each frame. The objects are people, objects, backgrounds, etc. that exist within the field of view. Furthermore, in this specification, the term "subject" is broadly interpreted and is not limited to specific tangible objects, but may also include scenes such as scenery (landscapes), dawn and night, events such as travel and weddings, themes such as cooking and hobbies, and patterns.

[0028] Video data has a file format that corresponds to its data structure. The file format includes the file format corresponding to the video data codec (compression technology) and version information. File formats include Moving Picture Experts Group (MPEG)-4, H.264, Motion JPEG (MJPEG), High Efficiency Image File Format (HEIF), Audio Video Interleave (AVI), QuickTime file format (MOV), Windows Media Video (WMV), and Flash Video (FLV). MJPEG is a file format in which the frame images that make up a video are in JPEG (Joint Photographic Experts Group) format.

[0029] The file format is reflected in the data structure of each frame. In one embodiment of the present invention, the first data in the data structure of each frame begins with a start-of-image (SOI) marker segment or a bitmap file header, which is header information. This information includes, for example, information indicating the frame number (a serial number assigned in order from the frame at the start of shooting).

[0030] The data structure of each frame also includes frame image data. The frame image data indicates the resolution of the frame image recorded at the angle of view at the time of shooting, and the gradation values ​​of two colors (black and white) or three colors (RGB (Red, Green, Blue)) specified for each pixel. The angle of view is the range in data processing in which the image is displayed or drawn, and this range is specified in a two-dimensional coordinate space with two mutually perpendicular axes as coordinate axes.

[0031] The data structure of each frame may also include an area where additional information can be recorded (written). The additional information is tag information relating to each frame and the subject within each frame.

[0032] For example, if the video file format is HEIF, supplementary information in the Exif (Exchangeable Image File Format) format corresponding to each frame can be stored, specifically, information regarding the shooting date and time, shooting location, shooting conditions, etc. The shooting conditions include the type of shooting equipment used, exposure conditions such as ISO sensitivity, f-number, and shutter speed, as well as the content of image processing, etc. The content of image processing includes the name, characteristics, and equipment that performed the processing, as well as the area in the angle of view where the image processing was performed, of the image processing performed on the image data of the frame.

[0033] Additionally, the coordinate information of the focus point during recording of the video data or the coordinate information of the user's gaze position (the gaze position will be explained later) can be recorded as additional information in the video data file. The coordinate information is information that represents the coordinates of the focus point or the gaze position in a two-dimensional coordinate space that defines the angle of view of the frame.

[0034] [Regarding supplementary information] Each frame in video data is provided with a box area in which supplementary information can be recorded, and supplementary information related to the subject in the frame can be recorded. Specifically, items corresponding to the subject can be recorded as supplementary information related to the subject. Items are matters and categories to which the subject falls when classified from various perspectives, and are, more simply, words (phrases) that represent the type, state, nature, structure, attributes, and other characteristics of the subject. For example, in the case shown in Figure 2, "person," "woman," "Japanese," "carrying a bag," and "carrying a luxury bag" correspond to the items.

[0035] In addition, two or more items of incidental information may be added to one subject, or multiple items of incidental information with different levels of abstraction may be added. The more items of incidental information added to one subject, or the more specific (detailed) the incidental information, the higher the accuracy of the items of incidental information for that subject. Here, accuracy is a concept that represents the degree of detail (resolution) of the content of the subject described by the incidental information.

[0036] Furthermore, for a subject to which additional information of a certain item has been added, additional information of an item with higher accuracy than that item may be added. In the case shown in Fig. 2, for example, for a subject to which additional information of the item "person" has been added, additional information of the item "woman" with higher accuracy is added. Furthermore, for a subject to which additional information of the item "carrying a bag" has been added, additional information of the item "carrying a luxury bag" with higher accuracy is added. Note that the additional information is preferably defined for each hierarchy as shown in Fig. 3.

[0037] Furthermore, the subject items may include items that cannot be identified from the subject's appearance, such as the presence or absence of abnormalities such as disease in agricultural crops, or the quality of fruit such as its sugar content. Items that cannot be identified from the appearance as described above can be determined from the feature amounts of the subject in the image data. Specifically, the correspondence between the feature amounts of the subject and the subject's attributes is learned in advance, and the attribute of the subject can be determined (estimated) from the feature amounts of the subject in the image based on that correspondence.

[0038] The feature amount of the subject may be, for example, the resolution of the subject in the frame, the amount of data, the degree of blur, the degree of shake, the size ratio to the angle of view of the frame, the position in the angle of view, the color, or a combination of these. The feature amount can be calculated by applying a known image analysis technique to analyze the subject area in the angle of view. The feature amount may also be a value output by inputting a frame (image) into a mathematical model constructed by machine learning, and may be, for example, a one-dimensional or multidimensional vector value. Any value that is uniquely output when at least one image is input can be used as the feature amount.

[0039] In addition, the box region may record additional information indicating the position (coordinate position) of the subject within the angle of view and additional information indicating the distance to the subject in the depth direction (depth). The coordinates of the subject are the coordinates of a point on the edge of a region (hereinafter referred to as the subject region) that surrounds part or all of the subject in a two-dimensional coordinate space that defines the angle of view of the frame, as shown in FIG. 2 . The shape of the subject region is not particularly limited, but may be, for example, approximately circular or rectangular. The subject region may be extracted by a user specifying a certain range within the angle of view, or may be extracted automatically using a known subject detection algorithm or the like.

[0040] When the subject area is a rectangular area indicated by the dashed line in Fig. 2, the position of the subject is identified by the coordinates of two intersections (indicated by the white and black circles in Fig. 2) located at both ends of a diagonal line at the edge of the subject area. In this way, the position of the subject in the angle of view can be accurately identified by the coordinates of multiple points.

[0041] The subject area may also be an area specified by the coordinates of a base point within the subject area and the distance from the base point. For example, if the subject area is circular as shown in FIG. 4, the subject area is specified by the coordinates of the center (base point) of the subject area and the distance from the base point to the edge of the subject area (i.e., radius r). In this case, the coordinates of the center, which is the base point, and the radius, which is the distance from the base point, are the position information of the subject area. In this way, by using the base point within the subject area and the distance from the base point, the position of the subject can be accurately represented. Note that the position of a rectangular subject area may also be represented by the coordinates of the center of the area and the distance from the center in each coordinate axis direction.

[0042] Additionally, the box area may also contain additional information indicating the size of the subject (hereinafter referred to as size information). The size of the subject can be determined based on, for example, the position information of the subject, specifically, the position (coordinate position) of the subject in the angle of view, the depth of the subject, and the like.

[0043] Furthermore, as shown in FIG. 2 , the box area may also record additional information (hereinafter referred to as image quality information) that indicates the image quality of the subject. Image quality refers to the image quality of the subject indicated by the frame image data, such as the subject's resolution, noise, and brightness. Resolution includes the presence and degree of blur or blur, resolution, or a corresponding grade or rank. Noise includes the S / N value, the presence or absence of white noise, or a corresponding grade or rank. Brightness includes a luminance value, a score indicating brightness, or a corresponding grade or rank. Brightness may also include the presence or absence of exposure abnormalities such as blown-out highlights or blocked-up shadows (whether the range of brightness exceeds the range that can be expressed by gradation values). The image quality information may also include an evaluation result (sensory evaluation result) of the resolution, noise, brightness, etc., based on human sensibility.

[0044] Video data with the above-described additional information recorded in frames can be used for various purposes, such as creating training data for machine learning. More specifically, video data is annotated (selected) based on the additional information recorded for each frame, since the subject in the frame can be identified from the additional information (more specifically, the items of the additional information). The annotated video data and its frame image data are used to create training data, and the training data required for machine learning is collected and used to perform machine learning.

[0045] [Basic Flow of Recording Supplementary Information] Hereinafter, the basic flow of recording supplementary information for a frame in video data will be described with reference to FIGS.

[0046] When recording supplementary information for a frame, first, a subject within the frame (hereinafter referred to as a recognized subject) is recognized, as shown in Fig. 5. Specifically, a subject area is extracted within the angle of view of the frame, and the subject within the extracted area is recognized as the recognized subject. Note that if multiple subject areas are extracted within a frame, the same number of recognized subjects as the extracted areas are recognized.

[0047] Next, the recognized subject is set as a search subject. The search subject is a subject that is the target of a search process described later. If multiple recognized subjects are recognized, at least some of the multiple recognized subjects are set as search subjects.

[0048] Next, the incidental information that can be recorded for the search subject is searched for based on the search item. Note that recording incidental information for the search subject is synonymous with recording incidental information for the frame in which the search subject exists.

[0049] As shown in Fig. 6, the search items are a plurality of items (a group of items) set as candidates for additional information. For example, if the search subject is a person, the item "person" is searched for among the search items. The search items also include a plurality of items with varying degrees of precision (more specifically, levels of detail and abstraction) for a certain perspective (theme and category). For example, the search items include the item "person," and further include items representing gender, age, nationality, occupation, etc. as more detailed items related to "person."

[0050] Then, from the above search items, items that correspond to the search subject are searched for as incidental information that can be recorded for the search subject. At this time, the more items that are searched for, or the more specific (detailed) the items that are searched for, the higher the search accuracy.

[0051] Furthermore, the precision of the search items, i.e., the number and resolution of the search items, is variable and can be changed once set. For example, after the precision of the search items is set according to a first search subject, the precision of the search items used to search for incidental information for a second search subject can be changed according to the second search subject.

[0052] The precision of the search items may be set higher depending on the subject in the previous frame. For example, a search may be performed to determine whether a subject (first subject) in a certain frame is a person, and more precise search items such as gender, nationality, and age may be set for a subject (the same subject as the first subject) in a subsequent frame.

[0053] The method for searching for recordable incidental information for a search subject is not particularly limited. For example, the type, nature, and state of the subject may be estimated from the feature amount of the subject, and an item that matches or corresponds to the estimated result may be found among the search items. Furthermore, when multiple search subjects are set, recordable incidental information for each search subject is searched for separately for each search subject.

[0054] Next, based on the search results, the searched items (i.e., a portion of the search items) are recorded as incidental information in the frame in which the search subject is located. Recording incidental information in a frame means writing the incidental information in a box area provided in the image data of the frame. If no items corresponding to the search subject are found in the search items, incidental information indicating "no matching items" may be recorded in the frame in which the search subject is located. Furthermore, when multiple subjects are set as search subjects, as shown in FIG. 5 , the incidental information (items) is searched for for each subject, and the retrieved incidental information (items) is associated with a corresponding subject and recorded in the frame. Note that the search for incidental information (items) does not need to be performed for all of the multiple subjects in the frame.

[0055] When recording supplementary information for frames in video data using the above-described procedure, it is necessary to be able to efficiently search for supplementary information that can be recorded for a search subject from among search items (lists).

[0056] On the other hand, for example, as shown in Fig. 7, during recording of moving image data, the subject within a frame may change due to a scene change or the subject moving, etc. Also, there may be cases where multiple different subjects exist within the same frame. Naturally, the incidental information that can be recorded for a subject (search subject) may change depending on the subject.

[0057] Therefore, the search items that are the search range for the incidental information must be set appropriately depending on the subject. For example, if the search subject is a "person" or a "landscape," the incidental information (items) to be searched will be different, so the search items must be set taking this into consideration.

[0058] Furthermore, for the purpose of performing a highly accurate search for important subjects (main subjects), it is preferable to set highly accurate search items, for example, search items including a large number of detailed items. Furthermore, it is difficult and inefficient to record all applicable items for the subject in each of multiple frames in video data. Taking these points into consideration, it is necessary to set search items appropriately.

[0059] Therefore, in one embodiment of the present invention, the recording device and recording method described below are used from the viewpoint of appropriately recording supplementary information for frames in video data. The following describes the configuration of the recording device according to one embodiment of the present invention and the flow of the recording method according to one embodiment of the present invention.

[0060] [Configuration of a Recording Device According to an Embodiment of the Present Invention] A recording device according to an embodiment of the present invention (hereinafter referred to as recording device 10) is a computer including a processor 11 and a memory 12, as shown in Fig. 8. The processor 11 is configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a tensor processing unit (TPU). The memory 12 is configured, for example, by semiconductor memory such as a read-only memory (ROM) and a random access memory (RAM).

[0061] The recording device 10 also includes an input device 13 that accepts user operations, such as a touch panel and cursor buttons, and an output device 14 that accepts a display and a speaker. The input device 13 may include a device that accepts voice input from the user. In this case, the recording device 10 may recognize the user's voice, analyze the voice using morphological analysis, and acquire the analysis results as input information.

[0062] The memory 12 also stores a program for recording supplementary information for frames in moving image data (hereinafter referred to as a recording program). The recording program is a program for causing a computer to execute each process included in the recording method of the present invention (specifically, each step in the recording flow shown in FIG. 14). The recording program may be obtained by reading it from a computer-readable recording medium, or by downloading it via a communication network such as the Internet or an intranet.

[0063] The recording device 10 can also freely access various data stored in the storage 15. The data stored in the storage 15 includes data necessary for the recording device 10 to record the additional information, specifically, the data for the search items described above. The storage 15 may be built into or external to the recording device 10, or may be configured as a NAS (Network Attached Storage) or the like. Alternatively, the storage 15 may be an external device, such as online storage, that can communicate with the recording device 10 via the Internet or a mobile communication network.

[0064] In one embodiment of the present invention, the recording device 10 is configured to record moving image data and is configured, for example, by a moving image capture device such as a digital camera or video camera. The configuration (particularly the mechanical configuration) of the capture device that constitutes the recording device 10 is substantially the same as that of known devices with a moving image capture function. The capture device may also have an autofocus (AF) function that automatically focuses on a predetermined position within the angle of view. Furthermore, the capture device may have a function that uses the AF function to specify the focus position, i.e., the AF point, while recording the moving image data.

[0065] The above-described photographing device also has a function for detecting blurring of the angle of view caused by camera shake and blurring of the subject caused by the subject's movement. Here, "blur" refers to irregular and slow shaking (shake) and is different from, for example, an intentional change in the angle of view, specifically, an operation of quickly changing the orientation of the photographing device in a predetermined direction (specifically, a panning operation). Note that subject blurring can be detected, for example, by known image analysis techniques. Blurring of the angle of view can be detected, for example, by known shake detection devices such as a gyro sensor.

[0066] The image capture device may also include a viewfinder, specifically an electronic or optical viewfinder, through which a user (i.e., a person recording the video) looks while the video data is being recorded. In this case, the image capture device may have a function for detecting the positions of the user's line of sight and pupils while the video data is being recorded, and identifying the user's line of sight. The user's line of sight corresponds to the intersection of the line of sight of the user looking into the viewfinder and a display screen (not shown) within the viewfinder.

[0067] The image capturing device may also be equipped with a known distance sensor such as an infrared sensor, in which case the image capturing device can measure the distance (depth) in the depth direction for each subject within the angle of view.

[0068] The functions of the recording device 10, particularly the functions related to recording of incidental information into frames, will be described with reference to Figure 9. As shown in Figure 9, the recording device 10 has an acquisition unit 21, an input reception unit 22, a recognition unit 23, an identification unit 24, a search unit 25, a setting unit 26, a selection unit 27, and a recording unit 28. These functional units are realized by cooperation between the hardware devices (processor 11, memory 12, input device 13, and output device 14) included in the recording device 10 and software including the recording program described above. Each of the functional units described above will be described below.

[0069] The acquisition unit 21 acquires moving image data composed of a plurality of frames. Specifically, the acquisition unit 21 acquires moving image data by recording frames (frame images) at a constant frame rate within the field of view of the imaging equipment constituting the recording device 10.

[0070] (Input Receiving Unit) The input receiving unit 22 executes a receiving step, and in the receiving step, receives user operations performed in connection with recording of incidental information into a frame. The user operations received by the input receiving unit 22 include user inputs related to items of incidental information (hereinafter, "item inputs"). The item inputs are input operations performed to record incidental information corresponding to the items entered by the user.

[0071] To be more specific, for example, a predetermined item (attached information) is assigned to a button (e.g., one function key) selected by the user among the input devices 13 of the recording device 10. The operation of pressing this button is item input, and the item assigned to this button corresponds to the input item. However, item input is not limited to the above operation, and may also be, for example, voice input, in which the user pronounces the predetermined item.

[0072] (Recognition Unit) The recognition unit 23 executes a recognition process, in which it recognizes multiple recognized subjects in multiple frames constituting the video image data. Specifically, in the recognition process, it extracts a subject area in the field of view of the frame and identifies the subject in the extracted subject area. Here, "multiple recognized subjects in multiple frames" encompasses both a collection of subjects recognized in each of multiple frames and multiple subjects recognized in a single frame. Note that the mode of recognizing multiple recognized subjects in multiple frames may include a mode in which there is a frame in the multiple frames in which no recognized subject is recognized.

[0073] (Identification Unit) The identification unit 24 identifies, for each frame, the position, size, and image quality of a recognized subject within the frame, the focus position (AF point), and the user's gaze position when using the viewfinder. The position of the recognized subject within the frame is the position (coordinates) of the subject area within the field of view, the position in the depth direction (depth), or a combination of these. The position of the subject area (coordinate position in two-dimensional space) can be identified using the procedure described above, and the depth can be measured using a known distance sensor such as an infrared sensor. The size of the recognized subject within the frame can be identified from the position of the subject area within the field of view and the depth of the recognized subject. The image quality of the recognized subject within the frame can be determined by, for example, the presence or absence of blur, shake, exposure abnormality, or a combination thereof. The image quality of these subjects can be identified using an image analysis function or sensor provided in the imaging equipment constituting the recording device 10. The focus position and the user's gaze position when using the viewfinder are positions set when recording video data and can be identified using an image analysis function or sensor provided in the imaging equipment constituting the recording device 10. The items identified for each frame by the identification unit 24 are recorded in a box area in the data structure of each frame.

[0074] Furthermore, for each recognized subject, the identification unit 24 can identify whether the recognized subject is moving, and if so, the direction of movement, etc., from multiple frames including the frame in which the recognized subject is present.

[0075] (Search Unit) The search unit 25 executes a search process for a search subject. The search subject is a part or all of the multiple recognized subjects recognized by the recognition unit 23. There is no particular limitation on which recognized subject is determined as the search subject, but for example, the search subject may be determined according to a predetermined criterion, or the search subject may be determined based on a user selection.

[0076] In one embodiment of the present invention, the search unit 25 executes the search process for search subjects selected based on at least one of a first condition and a second condition (corresponding to a predetermined condition) for executing the search process. By setting conditions for the search subjects for which the search process is executed in this way, it is possible to limit the subjects to be searched. As a result, the load of the search process can be reduced.

[0077] The first condition is a condition based on image quality information or size information of the search subject in the frame. The image quality information and size information are information indicating the image quality (specifically, the presence or absence of blur, shake, and exposure abnormality) and size identified by the identification unit 24 for the recognition subject corresponding to the search subject. Examples of search subjects that satisfy the first condition include search subjects whose degree of blur or shake is below a predetermined level, or search subjects whose size is below a predetermined size. Here, the predetermined level is, for example, the limit value of image quality that is acceptable for use as training data for machine learning (specifically, scene learning, etc.).

[0078] By setting the first condition, a certain level of image quality is ensured for the search subject on which the search process is executed. Therefore, if the search process is executed on a search subject that satisfies the first condition, more accurate (more reliable) search results can be obtained.

[0079] The second condition is a condition based on the focus position (AF point) set when recording the video data or the user's gaze position while recording the video data. The focus position and the user's gaze position are positions identified by the identification unit 24 for the frame in which the search subject exists. A search subject that satisfies the second condition is, for example, a search subject that exists within a predetermined distance from the focus position or the user's gaze position in the angle of view. Note that when determining whether the second condition is met, the depth of the search subject (more specifically, the depth measured by the identification unit 24 for the recognized subject that corresponds to the search subject) may be taken into consideration.

[0080] By setting the second condition, for example, the search process can be performed on a main search subject or a search subject that the user pays attention to. In other words, by performing the search process on a search subject that satisfies the second condition, it is possible to record supplementary information on a subject that is important to the user.

[0081] The first condition or the second condition may be used to set a priority when selecting a search subject for executing the search process from a plurality of subjects. For example, if there is an upper limit to the number of search subjects, a score may be calculated for each of a plurality of recognition subjects depending on whether the first condition or the second condition is met, and a subject with a higher score may be set as the search subject.

[0082] In the search process, the search unit 25 searches for recordable incidental information for the search subject based on the search items, specifically, searches for items corresponding to the search subject from among the search items. The search items used in the search process are set by the setting unit 26 or selected by the selection unit 27.

[0083] In one embodiment of the present invention, the interval between frames at which the search unit 25 executes the search process (the execution rate of the search process) can be changed depending on the search item used during the search process. For example, the search process is normally executed every frame or every few frames. In contrast, when a specific search item is used, the interval between frames at which the search process is executed can be made wider; in other words, the execution rate of the search process can be made slower than usual.

[0084] (Setting Unit) The setting unit 26 executes a setting step, and in the setting step, sets search items according to search subjects for which the search items are to be executed (i.e., search subjects that satisfy the first condition or the second condition). In addition, in the setting step when there are multiple search subjects, the setting unit 26 sets different search items for each search subject.

[0085] Specifically, a plurality of search items (a group of search items) are prepared in advance, and each search item is associated with a feature of the subject. The setting unit 26 selects a search item from the group of search items that corresponds to the feature of the search subject for which the search item is to be executed, thereby setting the search item to be used in the search process for that search subject. As described above, the feature of the subject can be calculated by analyzing the subject area in the angle of view using known image analysis technology, or can be output by inputting an image into a mathematical model constructed by machine learning.

[0086] Furthermore, the mode of setting different search items for each search subject may include a mode in which the same search item is set for some search subjects among multiple search subjects. Furthermore, different search items may include, for example, a case in which some of the items included in the search items are missing (lacking).

[0087] In one embodiment of the present invention, the setting unit 26 sets a priority for each search subject. The priority is determined based on the search subject's category, display size, position in the angle of view, distance from the focus position or the user's line of sight, depth, presence or absence of movement, presence or absence of change in state, etc. Specifically, if the search subject is a person, a higher priority is set than if the search subject is the background. Furthermore, a higher priority is set for a moving search subject than for a stationary search subject. Furthermore, the priority may be set by the user. Note that the mode of setting the priority for each search subject may include a mode in which there is a search subject for which no priority is set among the search subjects.

[0088] In the setting process, the accuracy of search items set for search subjects with higher priorities is set higher than the accuracy of search items set for search subjects with lower priorities. Explaining this with reference to Fig. 10, the number of search items for search subjects with higher priorities (people in Fig. 10) is greater than the number of search items for search subjects with lower priorities (cars in Fig. 10), and more detailed (specific) items are included.

[0089] In addition, in one embodiment of the present invention, in the setting process, when setting search items for a search subject in each frame, the precision of the search items is set according to the results of the search process for the previous frame (i.e., the past).

[0090] Referring to FIG. 11 , assume that, among multiple frames constituting video image data, a search subject in a first frame exists in a second frame, which precedes the first frame. In FIG. 11 , for example, the search subject, a "child," exists in three consecutive frames (#i to #i+2). Here, of the two frames, the later frame corresponds to the first frame, and the earlier frame corresponds to the second frame. In this case, in the setting process, the precision of the search items set for the search subject in the first frame is set higher than the precision of the search items set for the search subject in the second frame. For example, as shown in FIG. 11 , the search items for the search subject (child) in frame #i+1 include more search items and more detailed items than the search items for the same search subject in frame #i. Similarly, the search items for the search subject in frame #i+2 have higher precision than the search items for the same search subject in frame #i+1.

[0091] To explain another example, the setting unit 26 may set a search item that defines a rough classification of the subject, such as the search item L1 in FIG. 12 , at the beginning of recording of the video image data and immediately after a change in the shooting scene. In this case, for example, suppose that the item "person" is searched for in the search subject in a frame using the search item L1. In this case, a search item L2 with higher accuracy related to people is set in the next frame. Similarly, if the item "vehicle" is searched for in the search subject in a frame using the search item L1, a search item L3 with higher accuracy related to vehicles is set in the next frame. Furthermore, suppose that the item "child" is searched for in the search subject in a certain frame using the search item L2. In this case, a search item L4 with higher accuracy related to children is set in the next frame.

[0092] The selection unit 27 receives a selection operation from the user regarding a search item, and selects (selects) the search item selected by the user from the group of search items described above based on the received selection operation. The selection of the search item by the selection unit 27 is performed, for example, at a stage before the recording of the incidental information starts.

[0093] The search items selected by the selection unit 27 are used preferentially in the search process by the search unit 25. Specifically, when searching for recordable incidental information (items) for a search subject in each frame, the search unit 25 uses the search items set by the setting unit 26 according to the search subject. At this time, for example, if the user has previously selected search items related to trains via the selection unit 27, the search unit 25 executes the search process using the selected search items related to trains in addition to or instead of the search items set by the setting unit 26. As a result, when a train appears as a subject in a frame, the train can be used as a search subject to search for incidental information (items) from the search items related to trains.

[0094] 13, the execution rate of the search process using the search item selected by the selection unit 27 may be lower than the execution rate of the search process using the search item set by the setting unit 26. This allows the search process using the search item selected by the user to be executed at a relatively low execution rate, for example, once every few frames.

[0095] The recording unit 28 executes a recording process, and in the recording process, records at least a part of the search items as supplementary information based on the result of the search process. Specifically, the recording unit 28 records the items found for the search subject in the search process in a box area in the data structure of the frame in which the search subject exists.

[0096] Furthermore, in the recording step, the recording unit 28 records the coordinate position of the focus position or the user's gaze position as incidental information for the frame in which the focus position or the user's gaze position has been identified by the identification unit 24. This allows the incidental information recorded for the search subject in each frame to be associated with the focus position or the gaze position in each frame. This allows, for example, when performing machine learning for scene recognition using video data, the incidental information recorded for the search subject in each frame to be used in association with the focus position or the gaze position in that frame.

[0097] Furthermore, when the input receiving unit 22 executes the receiving step and receives the above-described item input, the recording unit 28 executes the recording step for the input frame. The input frame is a frame corresponding to the item input among multiple frames constituting the video data, and specifically, a frame recorded at the time the item input was received. The input frame may also include frames before or after the time the item input was received (for example, several frames before or after the frame at the time the input was received).

[0098] Furthermore, in the receiving step, additional information items different from the search items set by the setting unit 26 can be received. In other words, when inputting an item, the user can specify a user-specific item that is not included in the normal search items. Then, in the recording step for the input frame, additional information corresponding to the item input by the user is recorded. For example, when a function key for inputting an item is pressed, the recording unit 28 records additional information corresponding to the item pre-assigned to that function key in the input frame. Alternatively, when the user inputs a new item by voice to input an item, the recording unit 28 records additional information corresponding to the item input by voice in the input frame.

[0099] [Regarding a Recording Flow According to an Embodiment of the Present Invention] Next, a recording flow using the recording device 10 will be described. The recording flow described below uses the recording method of the present invention. In other words, each step in the recording flow described below corresponds to a component of the recording method of the present invention. Note that the flow below is merely an example, and unnecessary steps in the flow may be deleted, new steps may be added to the flow, or the execution order of two steps in the flow may be reversed, without departing from the spirit of the present invention.

[0100] The recording flow by the recording device 10 proceeds according to the flow shown in Figure 14, and each step (process) in the recording flow is executed by the processor 11 provided in the recording device 10. That is, in each process in the recording flow, the processor 11 executes the corresponding data processing defined in the recording program. Specifically, the processor 11 executes a recognition process in the recognition process, a search process in the search process, a setting process in the setting process, and a recording process in the recording process.

[0101] The recording flow is triggered by the start of recording of moving image data (S001). If the user makes a selection regarding a search item during the recording flow, the selection operation is accepted (S002). Note that this step S002 is omitted if there is no selection operation by the user.

[0102] In the recording flow, a recognition process, a setting process, a search process, and a recording process are performed for multiple frames constituting video image data. That is, the processor 11 recognizes multiple recognized subjects in multiple frames and searches for recordable incidental information for a search subject, which is some or all of the multiple recognized subjects, based on search items. Furthermore, if there are multiple search subjects, the processor 11 sets different search items for each search subject. Then, based on the search results, the processor 11 records at least some of the search items as incidental information for each frame.

[0103] The search process is not limited to being performed after the recognition process, but may be performed at the same time as the recognition process. Furthermore, the plurality of frames may include frames for which the recognition process is not performed. Furthermore, when different search items are set for each search subject, the same search items may be set for some search subjects.

[0104] To explain the recording flow in more detail, first, frame number #i (i is a natural number) is set to 1, and the subject to be recognized in frame #i is recognized (S003, S004).

[0105] Then, some or all of the recognized objects in frame #i are set as search objects, and it is determined whether the search objects satisfy the first or second condition described above (S005, S006). Specifically, it is determined whether the search process can be performed on the search objects based on image quality information indicating the degree of blur and shake of the search objects, and the presence or absence of exposure abnormalities. Alternatively, it is determined whether the search process can be performed on the search objects based on the positional relationship between the focus position or gaze position and the search objects. Note that in step S006, the first or second condition is used for the set search objects, but when setting a search object in step S005, it may also be used as a condition for selecting a search object from the recognized objects.

[0106] Furthermore, when there are multiple search subjects that satisfy the first or second condition in each frame, a priority is set for each search subject (S007, S008). At this time, the multiple search subjects may include a search subject for which no priority is set.

[0107] Next, search items are set according to the search subjects determined to satisfy the first or second condition (S009). If there are multiple search subjects (more specifically, search subjects that satisfy the first or second condition) in frame #i, search items are set in step S009 according to the priorities set in step S008. Specifically, search items with higher accuracy are set for search subjects with higher priorities than search items set for search subjects with lower priorities.

[0108] Next, recordable incidental information (items) for a search subject that satisfies the first or second condition is searched for based on the search items set in step S009 (S010). If there are multiple search subjects (more specifically, search subjects that satisfy the first or second condition) in frame #i, in step S010, incidental information for each search subject is searched for based on the search items set according to the priority of each search subject. Also, if a user's selection of a search item has been accepted in step S002, incidental information for the search subject is searched for based on the search item selected by the user as well as the search items set in step S009.

[0109] Thereafter, the incidental information (items) searched for in S010 is recorded in frame #i (S011). If incidental information for multiple search subjects is searched for in step S010, the incidental information for the multiple search subjects is recorded in frame #i in step S011. Furthermore, if the focus position or the user's gaze position in frame #i is identified, the coordinate information of that position is recorded in frame #i as incidental information.

[0110] Next, it is determined whether to end recording of the moving image data (S012), and if not, i is incremented (S013), and the process returns to step S004, and the series of steps from S004 onwards are repeated. When i is 2 or greater, steps S004 to S011 executed for frame #i are generally the same as the procedure described above.

[0111] On the other hand, in step S009 from the second time onwards, a search item with accuracy corresponding to the result of the search process for the previous frame (specifically, frame #i-1) is set. More specifically, if the search subject in frame #i is the same as the search subject in frame #i-1, the accuracy of the search item for the search subject in frame #i is set higher than the accuracy of the search item for the search subject in frame #i-1. By gradually increasing the accuracy of the search item in this way as the frames progress, for example, for a search subject appearing in two or more consecutive frames, the later the frame, the more detailed information can be recorded as supplementary information.

[0112] When the subject in the frame changes due to a scene change or the like, it is advisable to return the search items to search items with initial accuracy (for example, search items including roughly classified items).

[0113] Furthermore, while the video data is being recorded, the user can input items at any timing. When an item is input, the item input is accepted, and additional information corresponding to the item input by the user is recorded in the input frame (S014, S015). This makes it possible to accept the item input by the user, i.e., an item of additional information different from the search item set in step S009, and record that additional information in the input frame. As a result, items that the user specifies, such as special items such as technical terms, can be recorded as additional information.

[0114] The recording flow then ends when the recording of the video data has finished. As described above, in a recording flow according to one embodiment of the present invention, search items used when searching for incidental information recordable for a search subject are set for each search subject. This makes it possible to appropriately and efficiently record incidental information corresponding to the subject in each frame (strictly speaking, the search subject) for each frame of video data.

[0115] More specifically, since search items are set for each search subject, if the subject in the frame changes due to a scene change, for example, search items are set according to the changed subject. This makes it possible to appropriately search for recordable incidental information (items) for the search subject from the search items even after the scene has changed.

[0116] In one embodiment of the present invention, when there are multiple search subjects, priorities are set for the multiple search subjects, and more accurate search items are set for search subjects with higher priorities, which allows more detailed information (items) to be searched for subjects that are more important to the user, and also allows the searched information (items) to be recorded as incidental information.

[0117] Furthermore, in one embodiment of the present invention, search items selected by the user can be used in the search process. This allows the search items selected by the user to be used in addition to the search items set by the recording device 10 (i.e., automatically set items) to search for incidental information (items) that can be recorded on the searched subject. As a result, it becomes easier to reflect the user's intentions, etc. in the search for incidental information, and a recording method that is more preferable for the user is realized.

[0118] In one embodiment of the present invention, the scope in which the search process is performed is limited. Specifically, the search process is performed only on search subjects that satisfy a predetermined condition (specifically, a first condition or a second condition). By limiting the search subjects on which the search process is performed in this manner, the load associated with the search process can be reduced. Furthermore, since the number of search subjects for which incidental information is recorded is limited, the storage capacity of video data including incidental information can be further reduced.

[0119] <<Other Embodiments>> The embodiments described above are specific examples for clearly explaining the recording method, recording device, and program of the present invention, and are merely examples, and other embodiments are also possible.

[0120] (Regarding search subjects for which the search process is executed) In the above embodiment, the search process is not executed for search subjects whose blur or blur exceeds a predetermined level. However, if the search subjects are a main subject and its surrounding subjects, the search process may be executed for the search subjects even if they are slightly blurred or blurred. In this case, the accuracy of the search items used in the search process may be changed depending on the degree of blur and blur, and the greater the degree of blur and blur, the lower the accuracy of the search items. Furthermore, the depth of the search subject and the blur or blur of the search subject may be comprehensively assessed to determine whether or not to execute the search process for the search subject. Furthermore, the search subject for which the search process is executed may be specified by the user. That is, the search process may be executed for a search subject specified by the user among multiple search subjects, and additional information may be recorded based on the search results.

[0121] (Devices and equipment constituting the recording device of the present invention) In the above embodiment, the recording device of the present invention is a video recording device (i.e., a device that records video image data). However, this is not limited to this, and the recording device of the present invention may also be a device other than the recording device, for example, an editing device that acquires video image data from the recording device after the video has been recorded and performs data editing.

[0122] (Regarding the execution timing of the recognition process, search process, setting process, and recording process) In the above embodiment, the recognition process, search process, setting process, and recording process are executed for frames in the moving image data while the moving image data is being recorded. However, this is not limited to this, and the above-mentioned series of processes may be executed after the recording of the moving image data has finished.

[0123] (Variations of Data in Which Additional Information Is Stored) In the above embodiment, additional information about a subject in a frame is stored in a portion of the video data (more specifically, a box area in the data structure of the frame). However, this is not limited to this. As shown in FIG. 15 , the additional information may be stored in a data file separate from the video data. In this case, the data file in which the additional information is stored (hereinafter referred to as the additional information file DF) is linked to the video data MD containing the frame to which the additional information is added, and specifically includes an identification ID of the video data. Furthermore, as shown in FIG. 15 , the additional information file DF stores, for each frame, the frame number to which the additional information is added and the additional information about the subject in that frame. By storing the additional information in a data file separate from the video data as described above, the additional information can be appropriately recorded for frames in the video data while minimizing an increase in the volume of the video data. Note that the above-described method of recording additional information for each frame in the additional information file DF may include a mode in which frames containing no additional information are included among the multiple frames constituting the video data.

[0124] (Application to Image Data) In the above embodiment, an example was described in which collateral information is recorded for a frame in moving image data consisting of multiple frames. The present invention is also applicable to cases in which collateral information is recorded for image data including still image data. In other words, a recording method according to one embodiment of the present invention is a recording method for recording collateral information in image data, and includes the above-mentioned recognition step, search step, recording step, and setting step. Furthermore, when the image data is still image data, the recognition step recognizes multiple recognized subjects within the image data.

[0125] (Regarding Processor Configuration) The processor included in the recording device of the present invention includes various types of processors. The various types of processors include, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various types of processors also include a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array). Furthermore, the various types of processors include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processing.

[0126] Furthermore, one functional unit of the recording device of the present invention may be configured by one of the various processors described above. Alternatively, one functional unit of the recording device of the present invention may be configured by a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs, or a combination of an FPGA and a CPU. Furthermore, multiple functional units of the recording device of the present invention may be configured by one of the various processors, or two or more of the multiple functional units may be combined into a single processor. Furthermore, as in the above-described embodiment, one processor may be configured by a combination of one or more CPUs and software, and this processor may function as multiple functional units.

[0127] Furthermore, a processor may be used that realizes the functions of the entire system including the multiple functional units in the recording device of the present invention on a single IC (Integrated Circuit) chip, as typified by, for example, an SoC (System on Chip), etc. Furthermore, the hardware configuration of the various processors described above may be an electric circuit (Circuitry) that combines circuit elements such as semiconductor elements.

[0128] REFERENCE SIGNS LIST 10 Recording device 11 Processor 12 Memory 13 Input device 14 Output device 15 Storage 21 Acquisition unit 22 Input reception unit 23 Recognition unit 24 Search unit 25 Identification unit 26 Setting unit 27 Selection unit 28 Recording unit DF Additional information file MD Video image data

Claims

1. A recording method for recording supplementary information to a frame in moving image data composed of a plurality of frames, comprising the steps of: a recognition step of recognizing a plurality of recognition subjects in the plurality of frames; a search step of searching the additional information recordable for a search subject that is at least a part of the plurality of recognition subjects based on a search item; a setting step of setting different search items for each of the search subjects when there are a plurality of the search subjects; a recording step of recording at least a part of the search items as the additional information based on a result of the search step, In the setting step, a priority is set for each of the search subjects; A recording method, wherein the accuracy of the search items set for the search subjects having a higher priority is higher than the accuracy of the search items set for the search subjects having a lower priority.

2. The recording method according to claim 1 , wherein the searching step is executed for the search subjects selected according to a predetermined condition.

3. The recording method according to claim 2 , wherein the condition is based on image quality information or size information of the search subject in the frame.

4. 3. The recording method according to claim 2, wherein the condition is based on a focus position set in a recording device that records the moving image data, or a condition based on a user's line of sight position during recording of the moving image data.

5. The recording method according to claim 4 , wherein in the recording step, coordinate information of the focus position or the line of sight position is recorded as the additional information for the frame.

6. The recording method according to claim 1 , wherein the search step uses the search item selected by a user.

7. 2. The recording method according to claim 1, wherein in said setting step, the precision of said search items is set according to results of said search steps executed in the past.

8. 8. The recording method of claim 7, wherein, when the search subject in a first frame among the plurality of frames exists in a second frame that precedes the first frame, the setting process sets the accuracy of the search item set for the search subject in the first frame to be higher than the accuracy of the search item set for the search subject in the second frame.

9. The method further includes a step of receiving an input from a user regarding the items of the additional information, The recording method according to claim 1 , wherein the recording step is executed for an input frame corresponding to an input by the user among the plurality of frames, and the additional information corresponding to the input item is recorded.

10. 10. The recording method according to claim 9, wherein in the accepting step, an item of the additional information different from the search item set in the setting step can be accepted.

11. The recording method according to claim 1 , wherein the auxiliary information is stored in a data file different from the video image data.

12. A recording device including a processor, the recording device recording supplementary information for a frame in moving image data composed of a plurality of frames, The processor, A recognition process for recognizing a plurality of recognition subjects in the plurality of frames; a search process for searching the supplementary information recordable for a search subject that is at least a part of the plurality of recognized subjects based on a search item; A setting process for setting different search items for each of the search subjects when there are a plurality of the search subjects; a recording process for recording at least a part of the search items as the additional information based on a result of the search process.

13. 2. A program for causing a computer to execute each of the recognition step, the search step, the setting step, and the recording step included in the recording method according to claim 1.

14. A recording method for recording supplementary information to image data, comprising the steps of: a recognition step of recognizing a plurality of recognition objects in the image data; a search step of searching the additional information recordable for a search subject that is at least a part of the plurality of recognition subjects based on a search item; a setting step of setting different search items for each of the search subjects when there are a plurality of the search subjects; a recording step of recording at least a part of the search items as the additional information based on a result of the search step.