Information creation method and information creation device

Abstract

The present invention provides an information creation method and an information creation device for effectively creating additional information relating to the characteristics of each of a plurality of sounds included in sound data. An information creation method according to one embodiment of the present invention comprises: a first acquisition step for acquiring sound data including a plurality of sounds from a plurality of sound sources; a setting step of setting accuracy for the sound source or the sound; and a creation step for creating, on the basis of the accuracy, information relating to the characteristics of the sound as attached information of the moving image data corresponding to the sound data.

Description

Technical Field

[0001] One embodiment of the present invention relates to an information production method and an information production device. Background Art

[0002] When creating a moving image file including sound data including sound from a sound source and moving image data corresponding to the sound data, information related to the characteristics of the sound is sometimes created as incidental information of the moving image data. Examples of the characteristics of the sound include physical characteristics such as volume, amplitude, and frequency, the type of the sound source, and the result of determination based on the sound (for example, refer to Patent Document 1).

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2011 / 145249 Summary of the invention

[0006] Technical issues to be solved by the invention

[0007] There are many kinds of sounds (especially non-verbal sounds such as environmental sounds) recorded as sound data, and it is not necessary to uniformly produce accompanying information according to a unified standard for all kinds of sounds. Based on this, it is required to effectively produce accompanying information related to the characteristics of the multiple sounds included in the sound data.

[0008] An object of the present invention is to provide an information producing method and an information producing device for effectively producing supplementary information related to characteristics of each of a plurality of sounds included in sound data.

[0009] Means for solving technical problems

[0010] In order to achieve the above-mentioned purpose, the information production method involved in one embodiment of the present invention includes: a first acquisition step of acquiring sound data including multiple sounds from multiple sound sources; a setting step of setting the accuracy of the sound source or sound; and a production step of producing information related to the characteristics of the sound as accompanying information of dynamic image data corresponding to the sound data based on the accuracy.

[0011] Furthermore, in the setting step, the importance level may be set for the sound or the sound source, and the accuracy may be set according to the importance level.

[0012] Also, the sound may be non-verbal.

[0013] Furthermore, the information production method according to one embodiment of the present invention may include a second acquisition step of acquiring dynamic image data including a plurality of image frames. Furthermore, in the setting step, the accuracy of the sound source may be set according to whether the sound source exists within the field of view of a corresponding image frame among the plurality of image frames.

[0014] Furthermore, the information production method according to one embodiment of the present invention may include: a determination step: when the sound source does not exist within the field of view of the corresponding image frame, determining whether the sound satisfies a predetermined criterion. Furthermore, in the setting step, the accuracy of the sound when the predetermined criterion is satisfied may be set higher than when the predetermined criterion is not satisfied.

[0015] Furthermore, an information production method involved in one embodiment of the present invention may include: a change step, in which, when the sound meets a specified benchmark, the orientation of the imaging lens of the camera device is changed in such a way that the imaging lens is brought closer to the sound source, or the zoom ratio of the camera device is reduced in such a way that the sound source is included in the field of view of the image frame.

[0016] Furthermore, in the setting step, the accuracy for the sound source is set based on the result of image recognition related to the sound source in the corresponding image frame or device information about an imaging device that captures the image frame and the image frame being associated with each other.

[0017] Furthermore, in the setting process, the accuracy of the sound source may be set based on device information. In this case, the device information may be information related to the focus position of the camera in the image frame or the sight position of the user of the camera in the image frame.

[0018] Furthermore, in the creation step, information on whether or not a sound source exists within the field of view of the corresponding image frame may be created as supplementary information.

[0019] Furthermore, the information production method according to one embodiment of the present invention may include: an inspection step of inspecting whether the sound satisfies the inspection criterion when the accuracy corresponding to the importance satisfies a prescribed condition. In this case, in the production step, information related to the inspection result in the inspection step may be produced as supplementary information.

[0020] Furthermore, during the production process, reliability information related to the reliability of the inspection result may be produced as supplementary information.

[0021] Furthermore, in the production process, importance information related to the importance may be produced as supplementary information.

[0022] Furthermore, when the accuracy corresponding to the importance satisfies the first condition, in the creation step, onomatopoeia information in which the sound is converted into a text as an onomatopoeia may be created as the supplementary information.

[0023] Furthermore, when the accuracy corresponding to the importance satisfies the second condition, in the creation step, mimetic word information in which the state of the sound source in the corresponding image frame is converted into a text as a mimetic word may be further created as the supplementary information.

[0024] Furthermore, an information production device involved in one embodiment of the present invention includes a processor, which performs the following processing: acquiring sound data including multiple sounds from multiple sound sources; setting accuracy for the sound source or sound; and based on the accuracy, producing information related to the characteristics of the sound as accompanying information of a dynamic image corresponding to the sound data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is an illustration of a dynamic image file.

[0026] Figure 2 A diagram related to dynamic image data and sound data.

[0027] Figure 3 It is a diagram showing a configuration example of an information production device according to one embodiment of the present invention.

[0028] Figure 4 This is a diagram showing an example of supplementary information.

[0029] Figure 5 This is a diagram showing an example of the relationship between the viewing angle and the position of the sound source.

[0030] Figure 6 It is a diagram related to the order of determining the location of the sound source.

[0031] Figure 7 This is a diagram related to another example of the procedure for identifying the position of a sound source.

[0032] Figure 8 This is a diagram showing various information included in the incidental information of the sound.

[0033] Fig. 9A This is a diagram related to the function of the information production device involved in the first embodiment of the present invention.

[0034] Fig. 9B It is a diagram showing an example of the imaging device and the pan / tilt head according to the first embodiment of the present invention.

[0035] Fig.10 This is an illustration about onomatopoeia information.

[0036] Fig.11 This is an illustration about information about mimetic words.

[0037] Fig.12 It is a diagram showing the information creation flow involved in the first embodiment of the present invention.

[0038] Fig.13 This is a diagram showing an example of a specific scene in which a moving image file is created in the first embodiment.

[0039] Fig.14 It is a diagram showing the flow of the production process.

[0040] Fig.15 It means in Fig.13 An example of the accompanying information of the sound produced under the circumstances.

[0041] Fig.16 This is a diagram related to the function of the information production device involved in the second embodiment of the present invention.

[0042] Fig.17 It is a diagram showing the information creation flow involved in the second embodiment of the present invention.

[0043] Fig.18 This is a diagram showing an example of a specific scene in which a moving image file is created in the second embodiment.

[0044] Fig.19 It means in Fig.18 An example of the accompanying information of the sound produced under the circumstances. DETAILED DESCRIPTION

[0045] The specific embodiments of the present invention are described. However, the embodiments described below are only examples for facilitating understanding of the present invention and do not limit the present invention. The present invention can be changed or improved on the embodiments described below without departing from its purpose. Furthermore, the present invention includes its equivalents.

[0046] Furthermore, in this specification, the concept of “device” includes a single device that performs a certain function, and also includes a combination of multiple devices that exist independently of each other and cooperate (coordinate) to perform a certain function.

[0047] Furthermore, in this specification, "person" refers to a subject that performs a certain behavior, and this concept includes individuals, groups such as families, legal persons such as companies, and groups, etc.

[0048] Furthermore, in this specification, "artificial intelligence (AI)" refers to artificial intelligence that uses hardware resources and software resources to implement intelligent functions such as reasoning, prediction, and judgment. In addition, the algorithm of artificial intelligence is arbitrary, such as expert system, case-based reasoning (CBR), Bayesian network, or subsumption architecture.

[0049] <<About one embodiment of the present invention>>

[0050] One embodiment of the present invention relates to an information production method and information production device for producing supplementary information of dynamic image data included in a dynamic image file based on sound data included in the dynamic image file. In addition, one embodiment of the present invention also relates to a dynamic image file including the supplementary information.

[0051] like Figure 1 As shown, a moving picture file includes moving picture data, sound data and accompanying information. Examples of moving picture file types include MPEG (Moving Picture Experts Group)-4, H.264, MJPEG (Motion JPEG), HEIF (High Efficiency Image File Format), AVI (Audio Video Interleave), MOV (QuickTime fileformat), WMV (Windows Media Video), and FLV (FlashVideo).

[0052] The dynamic image data is obtained by a known camera device such as a video camera and a digital camera. The camera device captures the subject within the field of view and generates image frame data at a constant frame rate, such as Figure 2 As shown, dynamic image data consisting of a plurality of image frames is acquired. In one embodiment of the present invention, the subject recorded in each image frame includes a background (scenery).

[0053] In addition, if Figure 2 As shown, each image frame in the moving image data is assigned a frame number (in the figure, marked as #n: n is a natural number).

[0054] In one embodiment of the present invention, the conditions in which multiple sound sources emit sound are photographed to produce dynamic image data. In detail, at least one sound source is recorded in each image frame included in the dynamic image data, and multiple sound sources are recorded in the dynamic image data as a whole. The sound source is an object that emits sound, specifically, an animal, plant, machinery, equipment, musical instrument, tool, sirens and alarms, vehicles, natural objects (environments) such as mountains and seas, accidents such as explosions, and natural phenomena such as thunder or wind and rain. In addition, the sound source may include a person.

[0055] Sound data is data that records sound in a manner corresponding to dynamic image data. Specifically, the sound data includes sounds from multiple sound sources recorded in the dynamic image data. That is, the sound data is obtained by collecting sounds from various sound sources using a microphone built into or external to a camera device during the acquisition of dynamic image data (i.e., during shooting). In one embodiment of the present invention, the sounds included in the sound data are mainly non-verbal sounds, such as the working sounds of machines, the sounds of vehicles, the sounds of natural objects such as waterfalls, the calls of animals, the sounds of accidents, the sounds of natural phenomena, and noises. Furthermore, the sounds included in the sound data may include emotional sounds such as human laughter, crying, and surprise, as well as sounds caused by human actions.

[0056] In one embodiment of the present invention, the dynamic image data and the sound data are synchronized with each other, and the acquisition of the dynamic image data and the sound data starts at the same time and ends at the same time. That is, in one embodiment of the present invention, the dynamic image data corresponding to the sound data is acquired during the same period as the acquisition period of the sound data.

[0057] The supplementary information is information related to the moving image data that can be recorded in the frame area set in the moving image file. The supplementary information includes tag information in the Exif (Exchangeable image file format) format, specifically, tag information related to the shooting date and time, shooting location, shooting conditions, etc.

[0058] And, if Figure 1 As shown, the incidental information involved in one embodiment of the present invention includes incidental information related to the image recorded in the dynamic image data (hereinafter referred to as incidental information of the image) and incidental information related to the sound included in the sound data (hereinafter referred to as incidental information of the sound). The incidental information of the image includes incidental information related to the subject in the image (hereinafter referred to as incidental information of the subject).

[0059] The supplementary information is described in detail in the following items.

[0060] The video file with the accompanying information can be used as teaching data in, for example, machine learning for voice recognition. Through this machine learning, a learning model (hereinafter referred to as a voice recognition model) that recognizes voice in an input video and outputs the recognition result can be constructed.

[0061] Furthermore, in one embodiment of the present invention, the sound data included in the dynamic image file includes one or more non-speech sounds. In this case, by using the dynamic image file as teacher data for machine learning, a sound recognition model for recognizing non-speech sounds and recognizing the type of sound can be constructed.

[0062] <<Configuration example of information creation device according to one embodiment of the present invention>>

[0063] like Figure 3 As shown, an information production device according to an embodiment of the present invention (hereinafter referred to as information production device 10 ) includes a processor 11 , a memory 12 , and a communication interface 13 .

[0064] The processor 11 is constituted by, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or a TPU (Tensor Processing Unit).

[0065] The memory 12 is composed of semiconductor memories such as ROM (Read Only Memory) and RAM (Random Access Memory). A program for creating incidental information of dynamic image data (hereinafter referred to as an information creation program) is stored in the memory 12. The information creation program is a program for causing the processor 11 to implement each step of the information creation method described later.

[0066] The information creation program may be obtained by reading it from a computer-readable recording medium, or by downloading it via a communication network such as the Internet or an intranet.

[0067] The communication interface 13 is constituted by, for example, a network interface card or a communication interface board, etc. The information production device 10 can communicate with other devices via the communication interface 13 and transmit and receive data between the devices.

[0068] like Figure 3As shown, the information production device 10 further includes an input device 14 and an output device 15. The input device 14 includes devices such as a touch panel and cursor buttons for receiving user operations and devices such as a microphone for receiving voice input. The output device 15 includes display devices such as a display and audio devices such as a speaker.

[0069] Furthermore, the information production device 10 can freely access various data stored in the storage device 16. The data stored in the storage device 16 includes data required for producing the supplementary information. Specifically, the storage device 16 stores data for specifying the sound source of the sound included in the sound data and data for identifying the subject recorded in the moving image data.

[0070] In addition, the storage device 16 may be built-in or external to the information production device 10, or may be composed of a NAS (Network Attached Storage), etc. Alternatively, the storage device 16 may be an external device that can communicate with the Internet or a mobile network through the information production device 10, such as an online storage device.

[0071] In one embodiment of the present invention, Figure 3 As shown, the information production device 10 is mounted on a camera or other imaging device. The mechanical structure of the imaging device (hereinafter referred to as the imaging device 20) equipped with the information production device 10 is substantially the same as that of a known imaging device having the function of acquiring dynamic image data and sound data. In addition, the imaging device 20 has an internal clock and has the function of recording the time at each time point during shooting. Thus, the shooting time of each image frame of the dynamic image data can be determined and recorded.

[0072] In addition, the imaging device 20 may be a portable device such as a digital camera, or may be a device that is fixed at a predetermined position for use such as a surveillance camera or a fixed-point camera.

[0073] like Figure 3 As shown, the camera device 20 includes an imaging lens 20L, and an image of a subject within the field of view is formed by the imaging lens 20L, and an image frame recording the subject is produced at a constant frame rate, thereby acquiring dynamic image data. In addition, the camera device 20 collects sounds from sound sources around the device through a microphone or the like during shooting to acquire sound data. Furthermore, the camera device 20 produces supplementary information based on the acquired dynamic image data and sound data, and produces a dynamic image file including the dynamic image data, sound data, and supplementary information.

[0074] The camera device 20 may have an auto focus (AF) function for automatically focusing on a predetermined position within the field angle during shooting, and a function for determining a focal position (AF point). The AF point is determined as a coordinate position when a reference position within the field angle is used as an origin. The field angle is a range in data processing for displaying or drawing an image, and its range is defined in a two-dimensional coordinate space with two axes orthogonal to each other as coordinate axes.

[0075] like Figure 3 As shown, the camera device 20 may include a viewfinder 20F that is peeked at by the user (i.e., the photographer) during shooting. In this case, the camera device 20 may have a function of detecting the line of sight and the position of the pupil of the user using the viewfinder to determine the line of sight position of the user. The line of sight position of the user is equivalent to the intersection position of the line of sight of the user peeking through the viewfinder 20F and the display screen in the viewfinder 20F.

[0076] The imaging device 20 may be equipped with a known distance sensor such as an infrared sensor. In this case, the distance (depth) of the subject within the field angle in the depth direction can be measured by the distance sensor.

[0077] The camera device 20 may be equipped with a sensor for GPS (Global Positioning System) or GNSS (Global Navigation Satellite System). In this case, the position (latitude and longitude) of the camera device 20 can be measured by the function of the sensor.

[0078] The camera device 20 can be used in a state supported by a pan / tilt during shooting (see Fig. 9B ). The gimbal may be a structure capable of changing the posture of the camera device 20, and may be provided with a posture changing mechanism and a control circuit thereof. In this case, the camera device 20 may communicate with the control circuit via the communication interface 13, and control the posture of the gimbal via the control circuit. Thus, during shooting, the direction of the imaging lens 20L may be changed according to the command signal from the camera device 20 to change the field of view.

[0079] <<About additional information>>

[0080] In one embodiment of the present invention, supplementary information of moving image data is produced by the function of the information production device 10 mounted on the imaging device 20. The produced supplementary information is added to the moving image data and the audio data to become a constituent element of the moving image file.

[0081] The supplementary information is generated by associating it with the image frame while the imaging device 20 is acquiring moving image data and audio data (that is, during shooting), for example.

[0082] In one embodiment of the present invention, the incidental information of the subject is generated based on the dynamic image data, and the incidental information of the sound is generated based on the sound data. The incidental information of the subject and the incidental information of the sound are generated by establishing associations with each other. Figure 4 As shown, each piece of incidental information is produced by establishing a corresponding association with two or more image frames among the plurality of image frames included in the dynamic image data. Specifically, incidental information of sound is produced by establishing a corresponding association with two or more image frames captured during the generation of the sound. Incidental information of the subject is produced by establishing a corresponding association with the incidental information of the sound and the two or more image frames established in a corresponding association.

[0083] In addition, information related to the correspondence between the incidental information of the sound and two or more image frames (hereinafter referred to as the correspondence information) can be prepared as the incidental information. The correspondence information is the time of the start time point and the end time point of the sound generation period, or Figure 4 Information related to the frame numbers of the image frames captured at the start time point and the end time point is shown.

[0084] The incidental information of the subject is information related to the subject existing within the field angle of the image frame constituting the dynamic image data, for example, Figure 4 As shown, the information related to the category of the object is included. The category of the object is a type based on the morphological attribute of the object, and specifically, it is a general name of an object or event.

[0085] The method of determining the type of the subject is not particularly limited, but the type of the subject can be determined based on the characteristics of the area where the subject exists in the image frame by using a known subject recognition technology and image processing technology. The characteristics of the area where the subject exists include the hue, saturation, brightness, shape, size, and position within the field of view of the area.

[0086] Furthermore, when the subject is a sound source and the specified conditions are met, as additional information of the subject, mimetic word information is generated in which the state of the subject as the sound source is textualized as a mimetic word. The mimetic word information is generated by determining the state of the subject based on the characteristics of the area where the subject exists in the image frame and by using a well-known subject recognition technology and image processing technology. The function of textualizing the state of the subject is realized by artificial intelligence (AI), specifically, by a learning model that outputs a mimetic word when the state of the subject is input.

[0087] Here, the state of the subject that is textualized as a mimetic word includes the subject's appearance, form (pattern), surface characteristics, posture, movement, state, speed, and expression. Furthermore, mimetic words include language that mimics the state of a person or object, specifically, mimetic words that express movement or appearance, mimetic words that express movement or state, and mimetic words that express expression or mood.

[0088] When the image frame memory contains a plurality of subjects, the supplementary information of the subject may be generated for each subject, or may be generated limited to a part of the subjects (for example, the main subject).

[0089] The incidental information of the sound is information related to the sound from the sound source stored in the dynamic image data, and in particular, information related to the non-verbal sound emitted by the sound source. The incidental information of the sound is generated each time the sound source emits a sound. In other words, Figure 4 As shown, for each of a plurality of sounds from a sound source included in the sound data, supplementary information of the sound is created for each sound.

[0090] like Figure 4 As shown, the incidental information of the sound includes feature information related to the characteristics of the sound. The characteristics of the sound are the nature and characteristics of the sound, the evaluation results of the sound, the language expressing the content of the sound, the effects and influences brought by the sound, and items related to other sounds. Specifically, the frequency, dominant frequency (resonant peak component), amplitude, volume (sound pressure), waveform of the sound, text information that textualizes the sound, and language (onomatopoeia) that imitates the sound are equivalent to the characteristics of the sound.

[0091] Furthermore, the category of sound and the type of sound source are equivalent to the characteristics of the sound. The category of sound indicates what kind of sound it is, whether it is noise, or what kind of scene the sound is in. The type of sound source is a type based on the morphological attributes of the sound source, specifically, a general name of an object, person, or event that emits the sound.

[0092] In one embodiment of the present invention, feature information is generated based on the accuracy set for the sound or the sound source that emits the sound. Accuracy is a concept that represents the degree of detail (precision) of information generated as feature information. For a sound or sound source with a higher accuracy, more detailed feature information is generated, for example, feature information related to more items is generated.

[0093] In addition, the concept of accuracy may include the choice of whether to generate feature information.

[0094] Furthermore, the accuracy is set according to the importance of the sound or the sound source. The importance can be expressed by stages or levels such as "high / medium / low", or can also be expressed by a numerical value.

[0095] The importance of a sound is the prominence of the sound, specifically, the degree to which the characteristics of the sound are prominent. The importance of a sound is set based on the physical properties of the sound, such as volume and frequency. For example, the louder the volume, the higher the importance is set.

[0096] Furthermore, the importance of a sound can be set according to the category of the sound. The category of a sound is a concept that indicates what kind of sound it is, for example, a sound that is emitted suddenly such as a warning sound, an environmental sound, a noise, or a heterogeneous and characteristic sound such as an explosion. For characteristic sounds, it is usually preferred to set the importance to a higher level, and for noise or environmental sounds, it is usually preferred to set the importance to a lower level. However, regarding environmental sounds, sometimes the sound source becomes the main subject (for example, the sound of a tram when shooting a tram), in which case, even for environmental sounds, the importance can be set to a higher level.

[0097] In addition, as a means of identifying the type of sound, AI for sound recognition can be used. Furthermore, the importance of sound can be set by AI for importance setting, specifically, by a learning model that outputs the importance of sound included in the sound data when sound data is input.

[0098] The importance of a sound source is the degree of influence of the sound source on the dynamic image data, and is set, for example, based on the sound data and the dynamic image data. Specifically, the importance of a sound source is set based on whether the sound source exists within the field of view of the corresponding image frame. Specifically, a sound source that exists as a subject within the field of view is selected as the object of the camera, and therefore has a high probability of being important to the user. Based on this situation, a higher importance is usually set for a sound source that exists within the field of view or a sound emitted from the sound source.

[0099] Regarding the method of determining whether the sound source exists within the field of view, Figure 5 The following is an example of the situation shown in the figure. Figure 5 In the case of explosions occurring outside the field of view, Figure 5 In FIG. 1 , the range surrounded by the dotted line represents the area of ​​the field angle (shooting field angle).

[0100] In order to determine whether a sound source exists within the field of view, first, the type of sound is determined from the sound data. Then, in the dynamic image data, it is determined whether a sound source corresponding to the type of sound exists in the subject shown in the image frame at the time when the sound is generated. Figure 5 In the case of, the sound source of the explosion sound (ie, the explosion site) is not recorded as a subject in the image frame and is therefore determined to be outside the field of view.

[0101] Furthermore, before specifying the type of sound from the sound data, it may be determined whether the subject (sound source) exists within the image frame at the time when the sound is generated.

[0102] In addition, the method for determining whether a sound source exists within the field of view angle is not limited to the above method. For example, the position of the sound source can be determined using a known sound source search technology, and whether the sound source exists within the field of view angle can be determined from the determined position of the sound source. In this case, a directional microphone is used as a sound collection microphone, and the position of the sound source is determined from the sound collection direction of the directional microphone to determine whether the sound source exists within the field of view angle. Furthermore, the directional microphone is preferably a microphone that combines multiple microphone elements to collect sounds over a wide range of 180° or more (preferably 360°), and is capable of determining the direction of each collected sound.

[0103] In the case where a sound source exists within the field of view of an image frame, the importance of the sound source can be set based on the result of image recognition related to the sound source in the image frame. Specifically, the importance can be set based on the size of the sound source relative to the field of view and the type of the sound source. In this case, for example, the larger the size, the higher the importance can be set. Furthermore, in the case where the sound source is a person, the importance can be set to be relatively low, and in the case where the sound source is an object, the importance can be set to be relatively high. In this way, by setting the importance of the sound source based on the result of image recognition related to the sound source, the appropriateness of the set importance is improved.

[0104] In addition, as a means of identifying the type of the sound source, AI for sound source identification may be used.

[0105] Furthermore, when there is a sound source within the field of view of the image frame, the importance of the sound source may be set based on the device information associated with the image frame for the camera 20 that captured the image frame. The device information is, for example, information on the focus position (AF point) of the camera 20 in the image frame or the line of sight position of the user of the camera 20 in the image frame (specifically, the line of sight position of the user detected by a viewfinder with a line of sight detection sensor).

[0106] When setting the importance based on the device information, the distance between the position of the sound source and the focal position or the distance between the position of the sound source and the line of sight position within the field of view is determined, and the smaller the distance, the higher the importance is set. This reflects that the closer to the focal position or the line of sight position, the more likely it is to be an important subject to the user. In this way, the importance is set based on the device information of the camera device 20, and the appropriateness of the set importance is improved. In particular, when the focal position of the camera device 20 or the line of sight position of the user is used as the device information, a more appropriate importance can be set due to the above reasons.

[0107] In addition, there is no particular limitation on the method of determining the position of the sound source within the field of view. Figure 6 As shown in FIG. 1 , in the field of view, a region surrounding a part or the whole of the sound source (hereinafter referred to as the sound source region) is defined. Then, if the sound source region is a rectangular region, two intersection points (at the ends of the diagonal lines) on the edge of the region can be defined. Figure 6 The coordinates of the points indicated by white circles and black circles in FIG. 1 are determined as the positions (coordinate positions) of the sound sources. Figure 7 As shown in FIG. 1 , when the sound source region is a circular region, the position of the sound source can be determined based on the coordinates of the center (base point) of the region and the distance from the base point to the edge of the region (i.e., radius r). In addition, when the sound source region is a rectangle, the position of the sound source can also be determined based on the coordinates of the center (intersection of the diagonal lines) of the region and the distance from the center to the edge.

[0108] Furthermore, when the importance is set according to the position of the sound source, the distance (depth) of the sound source may be referred to, and the importance may be set according to both the position and the distance of the sound source.

[0109] Furthermore, in the case where there is a sound source outside the field of view of the image frame, the importance of the sound source can be set based on the scene or event of the image recorded in the dynamic image data. Specifically, important scenes or events can be identified based on the dynamic image data, and the importance can be set based on the correlation between the identified scenes or events and the sounds emitted by the sound sources outside the field of view. For example, in a situation where a "festival" scene is recorded in the dynamic image data, the drum sound of the festival is recorded in the sound data. On the other hand, when the drum as its sound source is outside the field of view, the importance of the sound source (drum) is set higher. Furthermore, in the aforementioned Figure 5 In the scene shown, regarding a unique sound such as an explosion, the importance of the explosion sound is set high even if the sound source is outside the viewing angle.

[0110] Furthermore, when recognizing a scene or an activity, the location of the camera 20 may be determined by a GPS sensor included in the camera 20, and the scene or the activity may be estimated based on the location.

[0111] The method for setting the importance is not limited to the above method, and for example, the importance of the sound or the sound source may be specified by the user. In this way, the incidental information of the sound, more specifically, the accuracy of the incidental information when it is created, can reflect the user's intention.

[0112] If the importance of a sound or a sound source is set according to the above method, the precision corresponding to the importance is set for the sound or the sound source. Specifically, the higher the importance, the more detailed the precision is set. Then, for the sound or the sound source for which the precision is set, feature information about the sound is produced according to the precision. For example, for a sound with a higher importance, feature information is produced with a higher precision. On the contrary, for a sound with a lower importance, feature information is produced with a lower precision. As a result, for each of the multiple sounds included in the sound data, feature information can be produced more efficiently than when feature information is produced with a uniform precision or information amount.

[0113] Furthermore, by creating the feature information as incidental information, the feature information can be used when a dynamic image file including the incidental information is used as teacher data to implement machine learning. Specifically, when selecting (annotating) a dynamic image file as teacher data, the feature information included in the dynamic image file can be used.

[0114] Furthermore, by setting the accuracy of the sound included in the sound data or its sound source and creating feature information based on the accuracy, it is possible to more appropriately select the moving image files. That is, by creating feature information corresponding to the accuracy for the sound included in the sound data, for example, more detailed feature information is created for important sounds, so that annotations can be made based on the feature information.

[0115] And, if Figure 4 As shown, importance information related to the importance set for the sound or the sound source can also be created as incidental information of the sound. In this case, when the dynamic image file is used as the teacher data to implement machine learning, the importance information included in the dynamic image file can be used as the incidental information. Specifically, when implementing machine learning, data of sounds with higher importance can be extracted based on the importance information, and machine learning can be implemented using feature information on the sound.

[0116] Furthermore, when the importance is set for a sound source, if the position and distance (depth) of the sound source are determined, information on the position and distance (depth) of the sound source may be further created as incidental information of the sound.

[0117] Furthermore, it is possible to determine whether the sound source of the sound included in the sound data exists within the field of view of the image frame corresponding to the sound, such as Figure 4 As shown, information related to the determination result (hereinafter referred to as presence / absence information) is further prepared as incidental information of the sound. In this case, the moving image files used as teacher data can be filtered (annotated) based on the presence / absence information included in the moving image files.

[0118] And, if Figure 4 As shown, information related to the type of the sound source of the sound included in the sound data (hereinafter referred to as type information) can be created as the incidental information of the sound. In this case, the desired dynamic image file can be searched based on the type information included as the incidental information in the dynamic image file. That is, the type information as the incidental information can be used as a search keyword when searching for dynamic image files.

[0119] And, regarding the sounds or sound sources with high importance, e.g. Figure 8 As shown, onomatopoeia information that converts the sound into text as an onomatopoeia can be created as the accompanying information of the sound (more specifically, one of the above-mentioned characteristic information). The onomatopoeia information is created by applying a known sound recognition technology to the sound (non-verbal sound) included in the sound data and assigning the most appropriate language from the pronunciation of the sound. The function of converting the sound into text is realized by artificial intelligence (AI), more specifically, by a learning model that outputs an onomatopoeia when a sound is input.

[0120] In addition, onomatopoeia includes onomatopoeia (language that imitates speech sounds) such as human laughter and animal calls.

[0121] By creating onomatopoeia information that is textualized non-verbal sounds as supplementary information, the usefulness of the dynamic image file is further improved. That is, by implementing machine learning using the dynamic image file including onomatopoeia information as teacher data, it is possible to learn the relationship between non-verbal sounds and onomatopoeia information, thereby building a more accurate sound recognition model.

[0122] Furthermore, when onomatopoeia information is produced, information related to the type of onomatopoeia (e.g., human laughter, animal calls, etc.) can also be produced (see Fig.10 ). In addition, for sounds or sound sources with low importance, onomatopeia information may not be generated from the perspective of reducing the load, but the present invention is not limited to this and onomatopeia information may be generated.

[0123] In one embodiment of the present invention, Figure 8 As shown, the accompanying information of the sound may further include link destination information and rights relationship information.

[0124] The link destination information is information indicating a link to a storage destination (save destination) of the sound file when the sound data identical to the sound data of the moving image file is created as a separate file (sound file). In addition, since the sound data of the moving image file records sounds from a plurality of sound sources, a sound file may be created for each sound source. In this case, the link destination information is created for each sound file (i.e., for each sound source) as supplementary information.

[0125] The rights relationship information is information about the ownership of the rights related to the sounds included in the sound data and the ownership of the rights related to the dynamic image data. For example, when a dynamic image file is produced by shooting a scene in which multiple performers play in sequence, the rights (copyright) of the dynamic image data belong to the producer of the dynamic image file (i.e., the photographer). On the other hand, the rights related to the sounds (performance sounds) of the multiple performers recorded in the sound data belong to each performer or the group to which he or she belongs. In this case, the rights relationship information that specifies the ownership relationship of these rights is produced as incidental information.

[0126] <<About the functions of the information production device>>

[0127] refer to Fig. 9A , functions of an information production device 10 according to an embodiment of the present invention (hereinafter referred to as the first embodiment) are described.

[0128] like Fig. 9A As shown, the information production device 10 involved in the first embodiment includes an acquisition unit 21, a determination unit 22, a determination unit 23, a setting unit 24, a first production unit 25, a second production unit 26, and a change unit 27. These functional units are implemented by the cooperation of the hardware devices (processor 11, memory 12, communication interface 13, input device 14, output device 15 and storage device 16) of the information production device 10 and the software including the above-mentioned information production program. In addition, artificial intelligence (AI) is used to implement some functions. The following describes each functional unit.

[0129] (Acquisition Department)

[0130] The acquisition unit 21 controls each unit of the camera device 20 to acquire dynamic image data and sound data. In the first embodiment, the acquisition unit 21 synchronizes the dynamic image data and sound data and produces these data at the same time while multiple sound sources emit sounds (non-speech sounds). Specifically, the acquisition unit 21 acquires dynamic image data composed of multiple image frames in a manner that records at least one sound source in one image frame. In addition, the acquisition unit 21 acquires sound data including multiple sounds from multiple sound sources recorded in multiple image frames included in the dynamic image data. At this time, each sound is associated with two or more image frames acquired during the period when the sound is generated in the multiple image frames (for example, reference Figure 4 ).

[0131] (Confirmation Department)

[0132] The identification unit 22 identifies the content related to the sound included in the sound data based on the moving image data and the sound data acquired by the acquisition unit 21 .

[0133] Specifically, the identification unit 22 identifies the correspondence relationship between the sound and the image frame for each of the plurality of sounds included in the sound data, and more specifically, identifies two or more image frames acquired during the generation period of each sound.

[0134] Then, the identification unit 22 identifies the characteristics (volume, sound pressure, amplitude, frequency, type of sound, etc.) and the sound source of each sound.

[0135] Then, the determination unit 22 determines whether the sound source exists within the field of view of the corresponding image frame. Here, the corresponding image frame refers to an image frame captured at the time when the sound source emits the sound among the plurality of image frames included in the moving image data.

[0136] Furthermore, when the sound source exists within the field of view angle, the determination unit 22 determines the position and distance (depth) of the sound source within the field of view angle. In this case, the determination unit 22 recognizes an image (specifically, a sound source area) related to the sound source in the corresponding image frame, and determines the size and type of the sound source as a result of the image recognition. Furthermore, the determination unit 22 acquires device information related to the focus position (AF point) of the camera device 20 or the user's line of sight position in the corresponding image frame, and determines the distance (interval) between these positions and the position of the sound source.

[0137] (Judgment Department)

[0138] When the sound source does not exist within the viewing angle of the corresponding image frame, the determination unit 23 determines whether the sound from the sound source satisfies a predetermined criterion (hereinafter referred to as the determination criterion) based on the feature determined by the determination unit 22. The determination criterion is a criterion set for the sound from the sound source outside the viewing angle, such as whether the volume is above a certain level, whether the sound is in a certain frequency band, or whether the sound is a heterogeneous and characteristic sound.

[0139] In addition, the determination criterion may be set in advance on the imaging device 20 side, or may be set by the user.

[0140] (Settings)

[0141] The setting unit 24 sets the importance level for each of a plurality of sounds from a plurality of sound sources included in the sound data or for the sound source of each sound.

[0142] When setting the importance level for a sound source, the setting unit 24 sets the importance level according to whether the sound source exists within the viewing angle of the corresponding image frame.

[0143] If the sound source exists within the viewing angle, the setting unit 24 sets the importance of the sound source based on the result of image recognition related to the sound source in the image frame, that is, the size and type of the sound source specified by the specifying unit 22 .

[0144] Furthermore, when the sound source exists within the field of view, the setting unit 24 may determine the focus position of the camera 20 or the user's line of sight position based on the device information, and set the importance of the sound source based on the distance between the determined position and the position of the sound source.

[0145] Furthermore, when the sound source does not exist within the viewing angle, the setting unit 24 sets the importance of the sound from the sound source according to the determination result based on the determination unit 23. Specifically, the setting unit 24 sets a higher importance to the sound that satisfies the determination criterion than the case where the determination criterion is not satisfied. The importance of the sound from the sound source outside the viewing angle is usually set to be low, but when a characteristic sound such as an explosion satisfies the determination criterion, even the sound from the sound source outside the viewing angle may become an important sound for the user. In the first embodiment, considering this point, the importance of the sound from the sound source outside the viewing angle can be appropriately set according to the success or failure of the determination criterion.

[0146] Then, the setting unit 24 sets the accuracy for each sound or sound source according to the set importance. Specifically, the setting unit 24 sets a higher accuracy for a sound or sound source with a higher importance, and sets a lower accuracy for a sound or sound source with a lower importance.

[0147] (1st production department)

[0148] The first creating unit 25 creates feature information for each of the plurality of sounds from the plurality of sound sources included in the sound data, based on the feature determined by the determining unit 22. At this time, the first creating unit 25 creates the feature information based on the accuracy set by the setting unit 24 for the sound or its sound source, and specifically, creates the feature information with a detail level (fineness) corresponding to the accuracy.

[0149] Then, the first creating unit 25 creates correspondence information on the correspondence relationship between the sound and the image frame, based on the correspondence relationship between the sound and the image frame determined by the determining unit 22 .

[0150] Furthermore, the first creating unit 25 creates, as supplementary information of the sound, importance information on the importance set by the setting unit 24 to the sound or its sound source for each of the plurality of sounds from the plurality of sound sources included in the sound data.

[0151] Furthermore, when the determination unit 22 determines whether a sound source exists within the viewing angle of the corresponding image frame, the first creation unit 25 further creates presence information regarding the presence of the sound source within the viewing angle as supplementary information of the sound.

[0152] Further, when the type of the sound source within the field of view angle is determined by the determination unit 22, the first production unit 25 further produces type information related to the type of the sound source as additional information of the sound.

[0153] Further, when the accuracy set according to the importance of the sound or its sound source satisfies a specified condition (hereinafter referred to as the first condition), the first production unit 25 can produce onomatopoeia information obtained by texturizing the sound as an onomatopoeia as additional information of the sound. For example, as Fig.10 shown, when the sound included in the sound data is a dog bark and the accuracy set according to the importance of the sound satisfies the first condition, the first production unit 25 produces onomatopoeia information of "woof". At this time, as Fig.10 shown, the first production unit 25 can also produce additional information related to the type of the onomatopoeia.

[0154] The first condition corresponds to the accuracy for producing onomatopoeia information. For example, it corresponds to the accuracy corresponding to a certain level of importance or higher.

[0155] In addition, the sound may include a sound that lasts for a long time such as rain sound or a sound that repeatedly rings for a certain time such as an alarm sound. Regarding such a sound, it is not necessary to produce all onomatopoeia information during the generation period of the sound. For example, onomatopoeia information can be produced at a certain interval (specifically, at a frequency of once every several hundred frames). That is, the frequency of producing additional information can be changed according to the type of the sound (non-verbal sound).

[0156] (Second production unit)

[0157] The second production unit 26 produces additional information (additional information of the subject) for the subject existing within the field of view angle of the image frame included in the moving image data. In the first embodiment, when the accuracy corresponding to the importance set for the sound or the sound source satisfies a specified condition (hereinafter referred to as the second condition), the second production unit 26 produces mimicry word information for the sound source of the sound. The mimicry word information is information related to the state of the sound source in the corresponding image frame. For example, as Fig.11 shown, when the sound source within the field of view angle is a person smiling and the accuracy set according to the importance of the sound satisfies the second condition, the second production unit 26 produces mimicry word information of "hee hee". At this time, as Fig.11 shown, the second production unit 26 can also produce additional information related to the type of the human emotion texturized as a mimicry word.

[0158] In addition, as a method for producing mimicry word information for the state of the sound source, for example, a known image analysis technique can be used to determine the state of the sound source from the moving image, and an AI can be used to assign a mimicry word corresponding to the determined state.

[0159] The second condition is equivalent to the accuracy of the mimetic word information to be produced, for example, the accuracy corresponding to a certain level of importance or higher. The sound source whose accuracy satisfies the second text condition can be, for example, the main subject in the corresponding image frame. The main subject may be the subject with the largest size in the image frame or the subject closest to the focus position or the user's line of sight position.

[0160] As described above, according to the function of the second production unit 26, it is possible to produce mimetic word information that expresses the state of the sound source in language (mimetic word) as supplementary information. This further improves the usefulness of the dynamic image file. Specifically, by implementing machine learning with a dynamic image file including mimetic word information as teacher data, it is possible to construct a learning model that outputs mimetic words based on an image of a subject (specifically, a sound source) when the image is input.

[0161] Furthermore, when the sound source moves, the second creating unit 26 can detect the movement of the sound source from the moving image shown in the moving image data, and create mimetic word information indicating the movement as the supplementary information. In particular, when the movement of the sound source satisfies a predetermined condition, such as when the sound source as a subject in the moving image moves significantly, it is preferable to create mimetic word information.

[0162] (Change Department)

[0163] The change unit 27 controls the pan / tilt to change the orientation of the imaging lens 20L of the camera device 20, or changes the zoom magnification of the camera device 20. Specifically, as described above, when the sound source does not exist within the field of view of the corresponding image frame, the determination unit 23 determines whether the sound from the sound source satisfies the determination criterion. Then, when the sound satisfies the determination criterion, the change unit 27 changes the orientation of the imaging lens 20L in such a way that the imaging lens 20L approaches the direction of the sound source (i.e., in such a way as to face the sound source). Alternatively, the change unit 27 reduces the zoom magnification of the camera device 20 so that the sound source is included in the field of view of the image frame.

[0164] The pan head is not particularly limited as long as it is a structure capable of changing the orientation of the imaging lens 20L. As an example, Fig. 9B The pan-tilt platform 33 is a three-axis moving mechanism that can move the frame 32 that accommodates the main body of the camera device 20 in three-axis directions (roll, pitch, and yaw). The structure of the frame 32 and the pan-tilt platform 33 as a three-axis moving mechanism is a well-known structure, and as shown in FIG. Fig. 9B As shown, the camera device 20 , the frame 32 and the pan / tilt head 33 may be a structure covered by a dome-shaped cover 31 .

[0165] By the function of the above-mentioned changing unit 27, when a characteristic sound such as an explosion sound is generated, if the sound source is not present within the field of view of the corresponding image frame, the field of view can be changed to include the sound source. Thus, for the characteristic sound generated outside the field of view, the image of the sound source (the location where the sound is generated) can be recorded.

[0166] In addition, the orientation and zoom factor of the imaging lens 20L (in other words, the changed field angle) changed by the changing unit 27 may be maintained for a predetermined period, specifically, while a sound satisfying the determination criterion is generated. Furthermore, the orientation and zoom factor of the imaging lens 20L may be returned to the settings before the change after being maintained for a predetermined period.

[0167] <<About the information creation process according to one embodiment of the present invention>>

[0168] Next, the information production flow using the information production device 10 is described. In the information production flow described below, the information production method of the present invention is used. That is, each step in the information production flow described below corresponds to the constituent elements of the information production method of the present invention.

[0169] The following process is merely an example, and unnecessary steps in the process may be deleted, new steps may be added to the process, or the execution order of two steps in the process may be replaced without departing from the spirit of the present invention.

[0170] Each step (process) in the information production flow is executed by the processor 11 included in the information production device 10. That is, in each process in the information production flow, the processor 11 executes processing corresponding to each process in data processing specified by the information production program.

[0171] The information production process according to the first embodiment is as follows Fig.12 In this process, the moving image data and the sound data are acquired, and the accompanying information of the moving image data is created to create a moving image file.

[0172] In the information production flow, the processor 11 implements a first acquisition step ( S001 ) of acquiring sound data including a plurality of sounds from a plurality of sound sources and a second acquisition step ( S002 ) of acquiring moving image data including a plurality of image frames.

[0173] In addition, Fig.12 In the illustrated flow, the second acquisition step is performed after the first acquisition step. However, for example, when a moving image with sound is captured using the imaging device 20 , the first acquisition step and the second acquisition step are performed simultaneously.

[0174] During the execution of the first acquisition step and the second acquisition step, the processor 11 executes a determination step (S003) to determine the contents related to the sound included in the sound data, specifically, the correspondence between the sound and the image frame, the characteristics of the sound, the type of the sound, and the sound source.

[0175] In addition, in the determination step, it is determined whether the sound source exists within the field of view of the corresponding image frame. For the sound source existing within the field of view, the position and distance (depth) of the sound source within the field of view, the size and type of the sound source, etc. are further determined.

[0176] When there is a sound source within the field of view, device information related to the focus position (AF point) of the camera 20 or the user's line of sight position in the corresponding image frame is obtained, and the distance between the position represented by the device information and the position of the sound source is determined.

[0177] In the determination step, when it is determined that the sound source exists within the viewing angle of the corresponding image frame (YES in S004 ), the processor 11 proceeds to the setting step ( S008 ).

[0178] On the other hand, when the sound source does not exist within the field of view of the corresponding image frame (No in S004), the processor 11 performs a determination step (S005). In the determination step, it is determined whether the sound from the sound source outside the field of view satisfies the determination criteria based on the features determined in the determination step.

[0179] Then, if the sound satisfies the judgment criterion ("Yes" in S006), the processor 11 implements a change process (S007). In the change process, the orientation of the imaging lens 20L of the camera 20 is changed so as to be closer to the direction of the sound source, or the zoom factor of the camera 20 is reduced so as to include the sound source in the field of view of the image frame.

[0180] After executing the changing step, the processor 11 proceeds to the setting step ( S008 ).

[0181] In the setting step, the processor 11 sets the importance level for each of a plurality of sounds from a plurality of sound sources included in the sound data or for the sound source of each sound.

[0182] The importance is set according to the presence or absence of a sound source within the field of view of the corresponding image frame. In addition, for a sound source existing within the field of view, the importance is set according to the result of image recognition related to the sound source in the image frame (specifically, the size of the sound source and the type of the sound source, etc.). In addition, for a sound source existing within the field of view, the importance is set according to the distance between the focus position of the camera device 20 determined from the device information or the line of sight position of the user and the position of the sound source.

[0183] Regarding the sound source that is not within the viewing angle, the importance is set according to the determination result in the determination step, and when the determination criterion is satisfied, a higher importance is set than when the determination criterion is not satisfied.

[0184] Then, in the setting process, the accuracy corresponding to the set importance is set for each sound or sound source. Here, as described above, the importance with respect to the sound source existing in the field of view angle is set according to whether the sound source exists in the field of view angle of the corresponding image frame. Therefore, for the sound source existing in the field of view angle, the accuracy is set according to whether the sound source exists in the field of view angle.

[0185] Furthermore, for a sound source existing within the viewing angle, the importance is set based on the result of image recognition related to the sound source in the corresponding image frame and the focus position or the user's line of sight position indicated by the device information. Therefore, for a sound source existing within the viewing angle, the accuracy is set based on the result of image recognition related to the sound source and the device information.

[0186] On the other hand, for a sound source that is not within the viewing angle, the importance is set according to whether the sound satisfies the judgment criterion, and therefore, for a sound from a sound source outside the viewing angle, the accuracy is set according to whether the judgment criterion is successful or not. At this time, the accuracy of the sound when the judgment criterion is satisfied is set higher than when the prescribed criterion is not satisfied.

[0187] Regarding the process so far, Fig.13 The conditions shown are examples for specific explanation. Fig.13 In the situation shown, a moving image is shot with a waterfall as the subject, and thunder occurs outside the field of view at a certain point in time after the start of shooting (the point in time corresponding to #1000 in the frame number). Therefore, the sound data acquired during the shooting of the moving image includes the sound of the waterfall and the sound of thunder. The sound of the waterfall corresponds to image frames #1 to 999 in the moving image. On the other hand, the sound of thunder corresponds to image frame #1000 shot at the time when the thunder occurred.

[0188] For example, during the period corresponding to frame number #1 to #999, the source of the sound of the waterfall is the waterfall, such as Fig.13 As shown, it exists within the field of view of the corresponding image frame. Therefore, the importance of the waterfall as a sound source is set according to the size of the waterfall within the field of view, the focus position of the camera device, or the distance between the user's line of sight and the waterfall. Fig.13 In the illustrated situation, the size of the waterfall is larger than the reference size, so the waterfall is equivalent to the main subject, and therefore the importance of the waterfall is set high. As a result, the accuracy of the waterfall is set high according to the importance.

[0189] On the other hand, the thunder that is the source of the thunder sound generated at the time point corresponding to frame number #1000 is located outside the field of view of the corresponding image frame. Therefore, in the determination process, it is determined whether the thunder sound satisfies the determination criteria, for example, whether the volume of the thunder sound is above the reference value. Fig.13 In the situation shown, the thunder does not satisfy the judgment criterion, so the importance of the thunder is set to be low. As a result, the accuracy for thunder is set to be low according to the importance.

[0190] In addition, if the thunder meets the judgment criteria, the importance and accuracy are set higher than when the thunder does not meet the judgment criteria. In this case, a change process is performed to change the direction or zoom ratio of the imaging lens 20L so that the thunder as the sound source is located within the field of view angle from outside the field of view angle.

[0191] Returning to the description of the information production flow, after the setting process is performed, the processor 11 performs a production process (S009) for producing the supplementary information of the moving image data. Fig.14 In the production process, as the incidental information of the moving image data, incidental information of the sound and incidental information of the image are produced.

[0192] The incidental information of the sound is produced according to the content determined in the determination process. Specifically, in the production process, a step (S021) of producing characteristic information for each of the multiple sounds from the multiple sound sources included in the sound data is implemented. In this step S021, the characteristic information is produced according to the accuracy set in the setting process. That is, when the accuracy for the sound or the sound source is set to be high, more detailed characteristic information is produced for the sound. On the other hand, when a lower accuracy is set for the sound or the sound source, the characteristic information for the sound is produced with a lower level of detail, or the characteristic information is omitted.

[0193] In the creation step, a step (S022) of creating importance information related to the importance of the sound or sound source set in the setting step as supplementary information is also performed.

[0194] In addition, in the creation process, a step (S023) of creating presence / absence information on whether the sound source exists within the field of view of the corresponding image frame as supplementary information is performed.

[0195] Then, when the accuracy set according to the importance of the sound or its sound source satisfies the first condition ( S024 ), a step of creating onomatopoeia information in which the sound is texted as an onomatopoeia as supplementary information is implemented ( S025 ).

[0196] Furthermore, when the accuracy corresponding to the importance set for the sound source existing in the field of view of the image frame satisfies the second condition (S026), a step of creating mimetic word information in which the state of the sound source is converted into a text as a mimetic word is implemented as additional information (S027). In the creation process, other related information (specifically, corresponding information and category information, etc.) is further created (S028).

[0197] During the acquisition of the moving image data and the sound data (i.e., during the shooting of the moving image), steps S001 to S009 in the information production flow are repeatedly performed. Fig.15 As shown, the incidental information of the sound is created for each of the plurality of sound sources from the plurality of sound sources included in the sound data. Fig.15 Indicates the previous description Fig.13 The accompanying information of the sound produced according to the situation.

[0198] like Fig.15 As shown in FIG. 1 , the “sound of a waterfall” corresponding to the image frames #1 to #999 is set to have a higher importance than the waterfall as the sound source. Fig.15 As shown, feature information is created with higher accuracy. Specifically, information on the volume, the type of sound source, and the positional relationship between the sound source and the focal position is created as feature information.

[0199] On the other hand, a lower importance is set for the "thunder" corresponding to the image frame #1000. Fig.15 As shown, the accuracy when creating feature information is lower, and information indicating the type of sound is created, while feature information related to the volume and the type of sound source is not created.

[0200] Then, when the acquisition of these data is completed (S010), the information production process is terminated.

[0201] << About the second embodiment of the present invention >>

[0202] Sometimes, sound is used to check whether there is any abnormality in the inspection object, such as in a hammer inspection. When the dynamic image data and sound data of the inspection are obtained and a dynamic image file is created, information related to the inspection result based on the sound can be created as supplementary information of the sound. This structure is set as the second embodiment of the present invention, and the second embodiment is described below.

[0203] In the following description, the second embodiment will be described mainly with respect to points different from the first embodiment.

[0204] In the second embodiment, the plurality of sounds included in the sound data include sounds emitted from each of the plurality of inspection objects during inspection. That is, in the second embodiment, the plurality of sound sources include the plurality of inspection objects. The plurality of inspection objects may be a plurality of inspection objects, or may be a plurality of inspection locations set in one object (including structures such as buildings).

[0205] Hereinafter, a description will be given of a case where a plurality of inspection target products (products) are subjected to hammer inspection as an example.

[0206] In addition, in the hammer inspection, each of a plurality of inspection target products is sequentially transported to an inspection location one by one, and is subjected to the hammer inspection at the inspection location.

[0207] The camera device 20 provided with the information production device 10 captures the hammer inspection situation, and the sound generated during the inspection is collected by the microphone provided with the camera device 20. Thus, dynamic image data and sound data about the hammer inspection are obtained. The sound data includes a plurality of sounds, and the plurality of sounds include inspection sounds and conveying sounds. The inspection sound is the sound made by the inspection object being struck for inspection at the inspection location. The conveying sound is the working sound when the conveying device (not shown) is operated in order to replace the inspection object arranged at the inspection location.

[0208] The information production device 10 can determine the inspection object that is arranged at the inspection location and is inspected. Specifically, a storage element storing its identification information (ID) is installed on each inspection object, and a sensor (not shown) reads the identification information from the storage element of the inspection object arranged at the inspection location. The information production device 10 communicates with the above-mentioned sensor through the communication interface 13, thereby acquiring the identification information read by the sensor. Thus, the ID of the inspection object being inspected is determined by the information production device 10.

[0209] In addition, when the inspection objects are arranged in different places, the arrangement positions of the inspection objects can be determined by using the GPS function, and each inspection object can be identified. In addition, when the inspection object has identification information on the surface, the identification information can be identified by using the image authentication technology of the camera device 20, and each inspection object can be identified based on the identification information.

[0210] Then, the information production device 10 produces the additional information of the sound for the inspection sound and the transmission sound included in the sound data. Specifically, the information production device 10 sets the importance of the inspection sound and the transmission sound respectively, and then sets the accuracy corresponding to the importance. At this time, a higher importance is set for the inspection sound, and a lower importance is set for the transmission sound.

[0211] Then, the information generating device 10 generates supplementary information corresponding to the accuracy for each sound. Regarding the inspection sound, information on the result of the hammer inspection is generated as supplementary information (strictly speaking, the above-mentioned characteristic information). On the other hand, regarding the transmission sound, no information related to the inspection result is generated.

[0212] More specifically, in the second embodiment, Fig.16 As shown, the information production device 10 has the same functional units as those of the first embodiment, and also has an inspection unit 28. When the accuracy set for the sound included in the sound data (in detail, the accuracy corresponding to the importance) satisfies the prescribed conditions, the inspection unit 28 inspects whether the sound satisfies the inspection criterion. Specifically, when the accuracy of the sound is the accuracy set for the inspection sound, the inspection unit 28 inspects whether the sound satisfies the inspection criterion based on the characteristics of the sound (for example, frequency, etc.). The inspection criterion is a criterion for determining whether the inspection object product as the sound source of the sound (inspection sound) is good or not, for example, whether the inspection sound is an abnormal sound different from the sound of a normal product.

[0213] In addition, as a means for checking whether a sound satisfies the inspection criteria, an inspection AI, more specifically, a learning model that determines whether the sound satisfies the inspection criteria based on the characteristics of the input sound, can be used.

[0214] Then, in the second embodiment, the first creating unit 25 creates information related to the inspection result of the inspection unit 28 as feature information (accompanying information of the sound) for the sound whose accuracy corresponding to the importance satisfies the prescribed condition. In this case, the first creating unit 25 may create information related to the physical characteristics (e.g., frequency, volume, amplitude, etc.) of the inspection sound used when checking whether the sound satisfies the inspection standard as the accompanying information.

[0215] Furthermore, in the second embodiment, when the first producing unit 25 produces information related to the inspection result, reliability information related to the reliability of the inspection result can be produced as feature information. Reliability is an indicator indicating the accuracy or validity of the inspection result, and is expressed, for example, by a numerical value calculated according to a predetermined calculation formula, a grade or classification determined according to the numerical value, or an evaluation term used when evaluating reliability.

[0216] In addition, as a mechanism for evaluating the reliability of the inspection results, AI for reliability evaluation may be used, or more specifically, other AI for evaluating the accuracy or likelihood of the inspection results based on the inspection AI may be used.

[0217] Next, refer to Fig.17, the information production process involved in the second embodiment is described. The information production process involved in the second embodiment is substantially the same as that of the first embodiment. Specifically, during the implementation of the hammer inspection, the processor 11 implements a first acquisition step (S041) of acquiring sound data and a second acquisition step (S042) of acquiring dynamic image data. In this case, if Fig.18 As shown, the moving image data alternately records images of inspecting the inspection target product and images of conveying the inspection target product, and the sound data alternately records inspection sounds and conveying sounds.

[0218] During the implementation of the first acquisition process and the second acquisition process, the processor 11 implements a determination process (S043) to determine the correspondence between sound and image frames, the characteristics of the sound, the type and sound source of the sound, etc. as the content related to the sound included in the sound data.

[0219] In addition, in the determination step, it is determined whether the sound source of the sound exists within the field of view of the corresponding image frame. For the sound source existing within the field of view, device information related to the focus position (AF point) of the camera 20 or the user's line of sight position in the corresponding image frame is obtained, and the distance between the position indicated by the device information and the position of the sound source is determined.

[0220] Furthermore, when the sound is an inspection sound, in the identification step, the ID of the inspection target product that is the sound source is identified. Specifically, the ID is identified by acquiring identification information of the inspection target product from the sensor.

[0221] After the determination process is performed, the processor 11 performs a setting process (S044) to set the importance of each of the multiple sounds (i.e., the inspection sound and the transmission sound) included in the sound data. In addition, in the setting process, the accuracy corresponding to the set importance is set for each sound or sound source. At this time, a higher importance and accuracy are set for the inspection sound, and a lower importance and accuracy are set for the transmission sound.

[0222] Then, the processor 11 determines whether the accuracy of each of the plurality of sounds for which the accuracy is set in the setting process satisfies the prescribed condition, specifically, whether it is equivalent to the accuracy of the inspection sound (S045). Then, the processor 11 performs an inspection process on the sound whose accuracy satisfies the prescribed condition, i.e., the inspection sound (S046). In the inspection process, it is checked whether the inspection sound satisfies the inspection standard, specifically, whether the inspection sound is an abnormal sound different from the sound of a normal product.

[0223] Then, the processor 11 performs a step of creating supplementary information of the moving image data ( S047 ).

[0224] Specifically, in the creation process, for each of the multiple sounds from the multiple sound sources included in the sound data, the accompanying information of the sound including the characteristic information is created. More specifically, for the inspection sound with a higher accuracy setting, such as Fig.19 As shown in FIG. 1 , in addition to information related to the frequency of the sound, information related to the inspection result in the inspection process is produced as feature information. Fig.19 As shown, regarding the inspection sound, reliability information related to the reliability of the inspection result is further prepared as incidental information of the sound.

[0225] In addition, regarding the inspection object, its ID (identification information) is determined in the determination process, such as Fig.19 As shown, the accompanying information of the sound including the information on the inspection result and the reliability information is associated with the ID of the object to be inspected.

[0226] On the other hand, regarding the transmission sound where the accuracy is set lower, such as Fig.19 As shown, information on the frequency of sound, etc. is generated as feature information, and information on the inspection result is not generated.

[0227] The above series of steps are repeatedly performed while the acquisition of dynamic image data and sound data continues, that is, while the hammer test continues. Then, when the inspection of all the inspection objects is completed and the acquisition of data is completed (S048), the information production process ends.

[0228] In the second embodiment, through the above steps, information related to the inspection result based on the sound can be produced as the additional information (feature information), and a dynamic image file including the additional information can be produced. By using the dynamic image file as teacher data to implement machine learning, a learning model that outputs (estimates) the inspection result from the input inspection sound can be constructed.

[0229] Furthermore, by creating reliability information related to the reliability of the inspection results as supplementary information, the above-mentioned learning accuracy can be improved. Specifically, the dynamic image files can be screened (annotated) according to the reliability of the inspection results. Thus, machine learning can be implemented on the basis of ensuring the reliability of the inspection results to obtain more appropriate learning results.

[0230] <<Other implementation methods>>

[0231] The above-described embodiment is a specific example given for the purpose of explaining the information production method and the information production device of the present invention. It is only an example, and other embodiments are also conceivable.

[0232] (Regarding files recording moving image data and sound data)

[0233] In the above embodiment, a video camera 20 with a microphone is used to capture a moving image with sound, and the moving image data and the sound data are simultaneously acquired, and these data are included in a moving image file. However, this is not limited to this. The moving image data and the sound data may also be acquired by other devices, and each data may be recorded in another file. In this case, it is preferred to synchronize and acquire each of the moving image data and the sound data.

[0234] (About accuracy setting)

[0235] In the above embodiment, importance is set for the sound or its sound source included in the sound data, and the accuracy is set according to the importance. However, it is not necessary to set the importance, and the accuracy may be directly set according to the information related to the sound or the sound source.

[0236] (About the sound included in the sound data)

[0237] In the above-mentioned embodiment, the plurality of sounds included in the sound data may include sounds other than non-speech sounds, that is, speech sounds such as human conversation sounds. In this case, the accuracy of the incidental information (sound incidental information) produced for the speech sounds may be set according to the importance of the sound source of the speech sounds. However, with regard to speech sounds, for example, even when the sound source is outside the field of view, when the speech sounds are clear, the importance and accuracy may be set higher, and the accuracy may be set in a different way from the case of non-speech sounds.

[0238] (Regarding the structure of the information production device)

[0239] In the above-mentioned embodiment, the structure in which the information production device of the present invention is mounted on the camera device is described. That is, in the above-mentioned embodiment, the incidental information of the dynamic image data is produced by the camera device that obtains the dynamic image data and the sound data. However, it is not limited to this, and the incidental information can also be produced by a device different from the camera device, specifically, by a PC, a smart phone or a tablet terminal connected to the camera device. In this case, the dynamic image data and the sound data can be obtained by the camera device, and the incidental information of the dynamic image data (in detail, the incidental information of the sound) can be produced by a device different from the camera device. Alternatively, the incidental information can be produced after the dynamic image data and the sound data are obtained.

[0240] (About the structure of the processor)

[0241] The processor included in the information production device of the present invention includes various processors, including, for example, a general-purpose processor, namely, a CPU, which executes software (programs) to function as various processing units.

[0242] In addition, various processors include PLDs (Programmable Logic Devices) which are processors whose circuit configuration can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays).

[0243] Furthermore, various processors include processors such as ASIC (Application Specific Integrated Circuit) which have a circuit configuration specifically designed to execute specific processing, that is, dedicated circuits.

[0244] Furthermore, one of the various processors described above may constitute one functional unit of the information production device of the present invention. Alternatively, one functional unit of the information production device of the present invention may be constituted by a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs or a combination of FPGAs and CPUs.

[0245] Furthermore, the plurality of functional units of the information production device of the present invention may be constituted by one of various processors, or two or more of the plurality of functional units may be constituted by one processor.

[0246] Furthermore, as in the above-described embodiment, a combination of one or more CPUs and software may constitute one processor, and the processor may function as a plurality of functional units.

[0247] Furthermore, the following method may be adopted: for example, a processor that realizes the functions of the entire system including multiple functional units in the information production device of the present invention by using one IC (Integrated Circuit) chip, as represented by SoC (System on Chip). Furthermore, the hardware structure of the above-mentioned various processors may be a circuit (Circuitry) composed of circuit elements such as semiconductor elements.

[0248] Explanation of symbols

[0249] 10-information production device, 11-processor, 12-memory, 13-communication interface, 14-input device, 15-output device, 16-storage device, 20-camera device, 20L-imaging lens, 20F-viewfinder, 21-acquisition unit, 22-determination unit, 23-judgment unit, 24-setting unit, 25-first production unit, 26-second production unit, 27-changing unit, 28-inspection unit, 31-cover, 32-frame, 33-pan head.

Claims

1. A method for producing information, comprising: A first acquisition step of acquiring sound data including a plurality of sounds from a plurality of sound sources; A setting step of setting the accuracy of the sound source or the sound; and The creating step creates information related to the characteristics of the sound as incidental information of the moving image data corresponding to the sound data based on the accuracy.

2. The information production method according to claim 1, wherein: In the setting step, importance is set for the sound or the sound source, and the accuracy is set according to the importance.

3. The information production method according to claim 1 or 2, wherein: The sound is a non-verbal sound.

4. The information production method according to claim 1, comprising: The second acquisition step is to acquire dynamic image data including a plurality of image frames. In the setting step, the accuracy for the sound source is set according to whether the sound source exists within a field of view of a corresponding image frame among the plurality of image frames.

5. The information production method according to claim 1, comprising: a determination step of determining whether the sound satisfies a predetermined criterion when the sound source does not exist within the field of view of the corresponding image frame, In the setting step, the accuracy with respect to the sound when the predetermined criterion is satisfied is set higher than when the predetermined criterion is not satisfied.

6. The information production method according to claim 5, comprising: A changing process, when the sound satisfies the specified criteria, changes the orientation of the imaging lens of the camera device in such a way that the imaging lens approaches the direction of the sound source, or reduces the zoom ratio of the camera device in such a way that the sound source is included in the field of view of the image frame.

7. The information production method according to claim 1, wherein: In the setting step, the accuracy for the sound source is set based on a result of image recognition related to the sound source in the corresponding image frame or device information about an imaging device that captures the image frame and is associated with the image frame.

8. The information production method according to claim 7, wherein: In the setting step, the accuracy of the sound source is set based on the device information. The device information is information related to a focus position of the imaging device in the image frame or a sight line position of a user of the imaging device in the image frame.

9. The information production method according to claim 1, wherein: In the creation step, information on whether the sound source exists within a viewing angle of a corresponding image frame is created as the supplementary information.

10. The information production method according to claim 2, comprising: an inspection step of inspecting whether the sound satisfies an inspection criterion when the accuracy corresponding to the importance satisfies a predetermined condition, In the producing step, information related to the inspection result in the inspecting step is produced as the supplementary information.

11. The information production method according to claim 10, wherein: In the production step, reliability information related to the reliability of the inspection result is further produced as the supplementary information.

12. The information production method according to claim 2, wherein: In the creating step, importance information related to the importance is created as the supplementary information.

13. The information production method according to claim 2, wherein: When the accuracy corresponding to the importance satisfies a first condition, in the creating step, onomatopoeia information in which the sound is converted into a text as an onomatopoeia is created as the supplementary information.

14. The information production method according to claim 13, wherein: When the accuracy corresponding to the importance satisfies a second condition, in the creating step, mimetic word information in which the state of the sound source in the corresponding image frame is converted into a text as a mimetic word is further created as the supplementary information.

15. An information production device comprising a processor, The processor performs the following processing: acquiring sound data including a plurality of sounds from a plurality of sound sources; Setting accuracy for the sound source or the sound; and Based on the accuracy, information on the characteristics of the sound is created as incidental information of a moving image corresponding to the sound data.

Images