Information creation method, information creation device, and moving image file

Abstract

The present invention provides an information creation method and an information creation device for creating additional information useful in learning related to sound included in sound data, and a moving image file including the additional information. An information creation method according to one embodiment of the present invention comprises: a first acquisition step for acquiring sound data including a plurality of sounds from a plurality of sound sources; and a creation step for creating text information in which the speech sound is textualized and related information relating to the textualization of the sound as attached information of the moving image data corresponding to the sound data.

Description

Technical Field

[0001] One embodiment of the present invention relates to an information production method and an information production device for producing supplementary information of moving image data corresponding to the sound data based on the sound data. In addition, one embodiment of the present invention also relates to a moving image file including the supplementary information. Background Art

[0002] When using a moving image file having audio data including sound from a sound source, text information in which the sound included in the audio data is converted into text may be created as supplementary information of the moving image data corresponding to the audio data (for example, see Patent Document 1).

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2007-104405 Summary of the invention

[0006] Technical issues to be solved by the invention

[0007] As described above, text information converted from sound and a dynamic image file including text information are used, for example, for machine learning. In this case, the accuracy of learning may be affected by the accompanying information included in the dynamic image file. Therefore, it is required to provide a dynamic image file having accompanying information useful in the above learning.

[0008] One embodiment of the present invention is to solve the above-mentioned problems of the prior art, and an object of the present invention is to provide an information production method and an information production device for producing supplementary information useful for learning related to sound included in sound data.

[0009] Furthermore, an object of one embodiment of the present invention is to provide a moving image file including the above-mentioned supplementary information.

[0010] Means for solving technical problems

[0011] In order to achieve the above-mentioned purpose, an information production method involved in one embodiment of the present invention includes: a first acquisition step of acquiring sound data including multiple sounds from multiple sound sources; and a production step of producing text information that textualizes the sound and related information related to the textualization of the sound as accompanying information of dynamic image data corresponding to the sound data.

[0012] Also, the related information may include reliability information related to the reliability of the textualization of the sound.

[0013] Furthermore, the information production method may include a second acquisition step of acquiring dynamic image data including a plurality of image frames. In this case, in the production step, corresponding information indicating a corresponding relationship between two or more image frames among the plurality of image frames and text information may be produced as supplementary information. Furthermore, the text information may be information related to a phrase, clause or sentence that is a text of the sound, and the reliability information may be information related to the reliability of the phrase, clause or sentence with respect to the sound.

[0014] Furthermore, the information production method may include a second acquisition step of acquiring dynamic image data including a plurality of image frames. In this case, in the production step, sound source information related to the sound source and presence / absence information related to whether the sound source exists within the field of view of the image frame corresponding to the sound source may be produced as the supplementary information.

[0015] Furthermore, when the reliability of the text information is lower than a predetermined standard, in the creation step, alternative text information related to a text different from the text information may be created for the sound.

[0016] And, the related information may include error information related to a speaking error of a speaker as a sound source.

[0017] Furthermore, during the production process, reliability information may be produced based on the classification of the sound content.

[0018] Furthermore, in the creation process, sound source information related to the speaker as the sound source may be created. In this case, the related information may include degree information related to the degree of consistency between the speaker's mouth movement and the text information.

[0019] And, the related information may include speaking method information related to the speaking method of the sound.

[0020] Furthermore, in the production process, as text information, first text information in which the voice is converted into text while maintaining the language system of the voice and second text information in which the voice is converted into text by changing the language system can be produced. In this case, the related information may include language system information related to the language system of the first text information or the second text information or change information related to the change of the second text information to the language system.

[0021] Furthermore, the related information may include information related to the reliability of the second text information.

[0022] Furthermore, in the production process, text information and reliability information may be produced for each of the plurality of sounds. In this case, the information production method may further include a display process of displaying statistical data obtained by statistically processing the reliability information produced for each of the plurality of sounds.

[0023] Furthermore, the information production method may further include: an analysis step of analyzing the reason why the reliability becomes lower than a prescribed benchmark when the reliability indicated by the reliability information becomes lower than a prescribed benchmark; and a notification step of notifying the reason.

[0024] Furthermore, when information other than text information in the supplementary information is set as non-text information, the cause can be identified based on the non-text information in the analysis step.

[0025] Furthermore, the information production method may further include a determination step of determining whether there is a change in the audio data or the dynamic image data based on the movement of the mouth of the speaker of the audio in the text information and the dynamic image data.

[0026] Furthermore, the sound may be a speech sound.

[0027] Furthermore, an information production device involved in one embodiment of the present invention includes a processor, which performs the following processing: obtaining sound data including multiple sounds from multiple sound sources; and producing text information that textualizes the sounds and related information related to the textualization of the sounds as accompanying information of dynamic image data corresponding to the sound data.

[0028] Furthermore, a moving image file according to an embodiment of the present invention includes: sound data including a plurality of sounds from a plurality of sound sources; moving image data corresponding to the sound data; and

[0029] The accompanying information of the dynamic image data includes text information for converting the sound into text and related information related to the text conversion of the sound. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an illustration of a dynamic image file.

[0031] Figure 2 A diagram related to dynamic image data and sound data.

[0032] Figure 3 It is a diagram showing a configuration example of an information production device according to one embodiment of the present invention.

[0033] Figure 4 This is a graph related to incidental information of the sound.

[0034] Figure 5 This is a diagram related to the relevant information when the second text information is created.

[0035] Figure 6 It is a graph related to sound source information.

[0036] Figure 7It is a diagram related to the order of determining the location of the sound source.

[0037] Figure 8 This is a diagram related to another example of the procedure for identifying the position of a sound source.

[0038] Fig. 9 This is a diagram showing various information included in the incidental information of the sound.

[0039] Fig.10 This is a graph related to the mouth shape information and degree information included in the related information.

[0040] Fig.11 A diagram related to speaking method information included in the related information.

[0041] Fig.12 A diagram related to type information included in related information.

[0042] Fig.13 A diagram related to error information included in the related information.

[0043] Fig.14 This is a diagram related to the function of the information production device involved in one embodiment of the present invention.

[0044] Fig.15 This is a graph related to statistical data obtained by statistically processing incidental information of sound.

[0045] Fig.16 This is a diagram related to the main process in the information production process involved in one embodiment of the present invention.

[0046] Fig.17 It is a diagram showing the flow of the production process.

[0047] Fig.18 This is a diagram related to a sub-process in the information production process involved in one embodiment of the present invention.

[0048] Fig.19 is a graph representing a list of speakers. DETAILED DESCRIPTION

[0049] The specific embodiments of the present invention are described. However, the embodiments described below are only examples for facilitating understanding of the present invention and do not limit the present invention. The present invention can be changed or improved on the embodiments described below without departing from its purpose. Furthermore, the present invention includes its equivalents.

[0050] Furthermore, in this specification, the concept of “device” includes a single device that performs a certain function, and also includes a combination of multiple devices that exist independently of each other and cooperate (coordinate) to perform a certain function.

[0051] Furthermore, in this specification, "person" refers to a subject that performs a certain behavior, and this concept includes individuals, groups such as families, legal persons such as companies, and groups, etc.

[0052] Furthermore, in this specification, "artificial intelligence (AI)" refers to artificial intelligence that uses hardware resources and software resources to implement intelligent functions such as reasoning, prediction, and judgment. In addition, the algorithm of artificial intelligence is arbitrary, such as expert system, case-based reasoning (CBR), Bayesian network, or subsumption architecture.

[0053] <<About one embodiment of the present invention>>

[0054] One embodiment of the present invention relates to an information production method and information production device for producing supplementary information of dynamic image data included in a dynamic image file based on sound data included in the dynamic image file. In addition, one embodiment of the present invention also relates to a dynamic image file including the supplementary information.

[0055] like Figure 1 As shown, a moving picture file includes moving picture data, sound data and accompanying information. Examples of moving picture file types include MPEG (Moving Picture Experts Group)-4, H.264, MJPEG (Motion JPEG), HEIF (High Efficiency Image File Format), AVI (Audio Video Interleave), MOV (QuickTime fileformat), WMV (Windows Media Video), and FLV (FlashVideo).

[0056] The dynamic image data is obtained by a known camera device such as a video camera and a digital camera. The camera device captures the subject within the field of view and generates image frames at a constant frame rate, such as Figure 2 As shown, dynamic image data including multiple image frames is obtained. Figure 2 As shown, each image frame in the moving image data is assigned a frame number (in the figure, marked as #n: n is a natural number).

[0057] In one embodiment of the present invention, the situation where multiple sound sources emit sound is captured to produce dynamic image data. Specifically, at least one sound source is recorded in each image frame included in the dynamic image data, and multiple sound sources are recorded in the entire dynamic image data. Examples of multiple sound sources include multiple people having a conversation or meeting, or more than one person speaking and more than one object.

[0058] Sound data is data in which sound is recorded in a manner corresponding to the dynamic image data. Specifically, the sound data includes sounds from multiple sound sources recorded in the dynamic image data, and is obtained by collecting sounds from various sound sources by a microphone built into or external to the camera device during the acquisition of the dynamic image data (i.e., during shooting). In one embodiment of the present invention, the sounds included in the sound data are mainly speech sounds (voices), such as the sounds of people talking or dialogue. However, this is not limited to this, and the sounds may include, for example, the calls of animals, laughter, breathing, and other speech sounds made by humans, as well as sounds that can be expressed as onomatopeia (language expressed by imitating speech). Furthermore, the sounds included in the sound data may include noise and environmental sounds in addition to the main sounds such as speech sounds.

[0059] Furthermore, speech sound may include voice during singing and voice during speech or lines. In addition, hereinafter, a person who is the sound source of speech sound is also referred to as a "speaker".

[0060] In one embodiment of the present invention, the dynamic image data and the sound data are synchronized with each other, and the acquisition of the dynamic image data and the sound data starts at the same time and ends at the same time. That is, in one embodiment of the present invention, the dynamic image data corresponding to the sound data is acquired during the same period as the acquisition period of the sound data.

[0061] The supplementary information is information related to the moving image data that can be recorded in the frame area set in the moving image file. The supplementary information includes tag information in the Exif (Exchangeable image file format) format, specifically, tag information related to the shooting date and time, shooting location, shooting conditions, etc.

[0062] Furthermore, the supplementary information according to one embodiment of the present invention includes information on a subject recorded in moving image data and supplementary information on sound included in sound data.

[0063] The supplementary information is described in detail in the following items.

[0064] <<Configuration example of information creation device according to one embodiment of the present invention>>

[0065] like Figure 3 As shown, an information production device according to an embodiment of the present invention (hereinafter referred to as information production device 10 ) includes a processor 11 , a memory 12 , and a communication interface 13 .

[0066] The processor 11 is constituted by, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or a TPU (Tensor Processing Unit).

[0067] The memory 12 is composed of semiconductor memories such as ROM (Read Only Memory) and RAM (Random Access Memory). A program for producing incidental information of dynamic image data (hereinafter referred to as an information production program) is stored in the memory 12. The information production program is a program for causing the processor 11 to implement each step in the information production method described below.

[0068] The information creation program may be obtained by reading it from a computer-readable recording medium, or by downloading it via a communication network such as the Internet or an intranet.

[0069] The communication interface 13 is constituted by, for example, a network interface card or a communication interface board, etc. The information production device 10 can communicate with other devices via the communication interface 13 and transmit and receive data between the devices.

[0070] like Figure 3 As shown, the information production device 10 further includes an input device 14 and an output device 15. The input device 14 includes devices such as a touch panel and cursor buttons for receiving user operations and devices such as a microphone for receiving voice input. The output device 15 includes display devices such as a display and audio devices such as a speaker.

[0071] Furthermore, the information production device 10 can freely access various data stored in the storage device 16. The data stored in the storage device 16 includes data required for producing the supplementary information. Specifically, the storage device 16 stores data for specifying the sound source of the sound included in the sound data and data for identifying the subject recorded in the moving image data.

[0072] In addition, the storage device 16 may be built-in or external to the information production device 10, or may be composed of a NAS (Network Attached Storage), etc. Alternatively, the storage device 16 may be an external device that can communicate with the Internet or a mobile network through the information production device 10, such as an online storage device.

[0073] In one embodiment of the present invention, Figure 3 As shown, the information production device 10 is mounted on a camera or other imaging device. The mechanical structure of the imaging device (hereinafter referred to as the imaging device 20) equipped with the information production device 10 is substantially the same as that of a known imaging device capable of acquiring dynamic image data and sound data. In addition, the imaging device 20 has an internal clock and has a function of recording the time at each time point during shooting. Thus, the shooting time of each image frame of the dynamic image data can be determined.

[0074] The camera device 20 images a subject within the field of view through an imaging lens (not shown) and creates an image frame recording the subject at a constant frame rate, thereby acquiring dynamic image data. In addition, the camera device 20 collects sound (specifically, the speech of a speaker) from a sound source around the device through a microphone or the like during shooting to acquire sound data. Furthermore, the camera device 20 creates supplementary information based on the acquired dynamic image data and sound data, and creates a dynamic image file including the dynamic image data, sound data, and supplementary information.

[0075] The camera device 20 may have an auto focus (AF) function for automatically focusing on a predetermined position within the field angle during shooting, and a function for determining a focus position (AF point). The AF point is determined as a coordinate position when a reference position within the field angle is used as an origin. The field angle is a range in data processing for displaying or drawing an image, and its range is defined in a two-dimensional coordinate space with two axes orthogonal to each other as coordinate axes.

[0076] The camera device 20 may also include a viewfinder that the user (i.e., the photographer) looks into during shooting. In this case, the camera device 20 may have a function of detecting the line of sight and the position of the pupil of the user using the viewfinder to determine the line of sight position of the user. The line of sight position of the user is equivalent to the intersection position of the line of sight of the user observing the viewfinder and the display screen (not shown) in the viewfinder.

[0077] The imaging device 20 may be equipped with a known distance sensor such as an infrared sensor. In this case, the distance (depth) of the subject within the field angle in the depth direction can be measured by the distance sensor.

[0078] <<About additional information>>

[0079] In one embodiment of the present invention, supplementary information of moving image data is produced by the function of the information production device 10 mounted on the imaging device 20. The produced supplementary information is added to the moving image data and the audio data to become a constituent element of the moving image file.

[0080] The supplementary information is generated, for example, while the camera 20 is acquiring moving image data and audio data (that is, during shooting). However, the invention is not limited thereto, and the supplementary information may be generated after shooting is completed.

[0081] In one embodiment of the present invention, the incidental information includes information generated based on the sound data (hereinafter referred to as incidental information of the sound). The incidental information of the sound is information related to the sound from the sound source stored in the dynamic image data, specifically, information related to the sound (speech sound) emitted by the speaker as the sound source. The incidental information of the sound is generated each time the speaker emits a speech sound. In other words, Figure 4 As shown, for each of a plurality of sounds from a plurality of sound sources included in the sound data, the accompanying information of the sound is created for each sound.

[0082] like Figure 4 As shown, the accompanying information of the sound includes text information that converts the sound into text and corresponding information.

[0083] Textualization refers to natural language processing of sound, specifically, it refers to recognizing sounds such as speech sounds, analyzing the meaning of the language (words) represented by the speech sounds, and assigning appropriate words according to the meaning. And, text information is information created by textualization. More specifically, a sound including multiple words such as a conversation sound represents a phrase, a clause, or a sentence, so the text information becomes information related to the phrase, clause, or sentence that textualized its sound. That is, text information is information that is a book of the speaker's speech content, and as long as the text information is referred to, the meaning of the speech sound (speech content) uttered by the speaker can be easily determined.

[0084] In addition, a "phrase" refers to a word that is composed of two or more words, such as a noun and an adjective, and functions as a word class. Also, a "clause" refers to a word that is composed of two or more words and functions as a word class, including at least a subject and a verb. Also, a "sentence" is composed of one or more clauses, and the article is completed by a period.

[0085] The text information is created by the function of the information creation device 10 included in the camera 20. The texting function is realized by, for example, artificial intelligence (AI), specifically, a learning model that estimates a phrase, a clause, or a sentence from input sound and outputs text information.

[0086] In one embodiment of the present invention, a first text information is prepared as text information while maintaining the language system of the sound and converting the sound into text. The language system of the sound refers to the concept of indicating the classification of languages ​​(specifically, the types of languages ​​such as Japanese, English, and Chinese) and whether it is a standard language or a language variant (dialect, jargon, or colloquialism, etc.). Maintaining the language system of the sound means using the same language system as the language system of the sound. That is, for example, when the language system of the sound is the standard language of Japanese, the text information (first text information) is prepared by converting the sound into text in the standard language of Japanese.

[0087] The language system used when creating text information may be automatically set in advance by the camera 20, or may be specified by the user of the camera 20. Alternatively, the language system of the voice may be estimated based on the characteristics of the voice using artificial intelligence (AI).

[0088] The corresponding information is information related to the correspondence between two or more image frames among the plurality of image frames included in the dynamic image data and the text information. Specifically, the sound (speech sound) emitted by the speaker may last for a period of time equivalent to several frames. Figure 4 As shown, the text information of the texted sound is associated with two or more image frames captured when the sound is generated. In one embodiment of the present invention, corresponding information indicating the corresponding relationship between two or more image frames and text information is prepared as incidental information of the sound.

[0089] Specifically, if Figure 4 As shown, corresponding information related to the start time point and the end time point of the text-based sound generation period is generated. Alternatively, for each of the start time point and the end time point of the text-based sound generation period, information related to the frame number of the image frame captured at that time point can be generated as corresponding information.

[0090] A moving image file including text information and corresponding information as supplementary information can be used as teacher data in machine learning for speech recognition, for example. Through this machine learning, a learning model (hereinafter referred to as a speech recognition model) can be constructed that converts speech sounds in an input image into text and outputs it. The speech recognition model can be used, for example, as a tool for displaying subtitles on a screen when playing a moving image.

[0091] Furthermore, in one embodiment of the present invention, as text information, the language system can be changed together with the first text information that converts the sound into text while maintaining the language system of the sound included in the sound data to create the second text information. The second text information is information that converts the sound into text using a language system different from the sound included in the sound data, in other words, the language system of the sound uttered by the speaker is changed into another language system to create the second text information.

[0092] For example, when the voice included in the voice data is in Japanese, the text information (first text information) is created in Japanese. Figure 5 As shown, the second text information is created by translating the phrases, clauses or sentences with the same meaning as the text information into a language other than Japanese (for example, English). Furthermore, when the text information (first text information) is created based on a dialect used in a region of Japan, the second text information is created by converting the phrases, clauses or sentences with the same meaning as the text information into the standard language of Japanese.

[0093] The second text information is produced using AI different from the AI ​​used for producing the first text information, such as an AI for translation. The language system used for producing the second text information may be automatically specified in advance on the camera 20 side, or may be selected by the user of the camera 20. Furthermore, the second text information may be produced by converting the first text information. Alternatively, the second text information may be produced by directly converting the sound included in the sound data into text using the changed language system.

[0094] In one embodiment of the present invention, Figure 4 As shown, the incidental information of the sound includes reliability information as relevant information related to the textualization of the sound included in the sound data. The reliability information is information related to the reliability of the textualization of the sound, and is information related to the reliability of the text information. In addition, when the first text information and the second text information are produced as text information, the reliability information is produced as information related to the reliability of the first text information.

[0095] Reliability refers to the accuracy of textualizing sound, that is, the reliability of the (textualized) phrase, clause or sentence relative to the sound. Specifically, it is an index that indicates the accuracy (likelihood) or ambiguity of the assigned sentence or clause. Reliability is expressed, for example, by a numerical value calculated by AI taking into account the clarity and noise of the sound, a numerical value derived from a calculation formula that quantifies reliability, a grade or classification determined by the numerical value, or an evaluation term used when qualitatively evaluating reliability (specifically, "high, medium, low", etc.).

[0096] In addition, the reliability information is preferably calculated by AI or the like in conjunction with the text information of the sound data.

[0097] like Figure 4 As shown, reliability information is generated for each text information, that is, for each textualized sound. There is no particular limitation on the method of generating reliability information. For example, it can be generated using a textualized phrase, clause, or sentence, that is, an algorithm or learning model that calculates the accuracy of the output result. In addition, reliability information can be generated based on the clarity of the sound (speech sound) and the presence or absence of noise. In addition, reliability information can be generated based on the presence or absence of homonyms or words with similar pronunciations.

[0098] By using a dynamic image file including reliability information as incidental information as teacher data to implement machine learning, it is possible to improve learning accuracy, and in detail, it is possible to improve the accuracy of the above-mentioned speech recognition model. That is, learning accuracy may be affected by the reliability of the dynamic image file as teacher data, and in detail, it may be affected by the reliability of the text information. By creating reliability information related to the reliability of the text information as incidental information, the reliability of the text information can be considered when implementing machine learning.

[0099] Specifically, it is possible to filter (annotate) dynamic image files according to the reliability of text information. Furthermore, dynamic image files are weighted according to the reliability of text information. For example, for dynamic image files with low text information reliability, the weight is set lower. As a result, more appropriate learning results can be obtained.

[0100] Furthermore, in one embodiment of the present invention, for text information whose reliability is lower than a prescribed benchmark, such as Figure 4 As shown, alternative text information can be added to the sound. Alternative text information is information about a certain sound that is created as a replacement candidate when the reliability of the text information is lower than a predetermined standard and is related to text that is different from the text information. Figure 4 In the example, since the reliability of textualization is low, alternative text information "sansei shimasu" is produced for the text information "reflection (hanseishimasu)". Thus, even if the text information may be wrong, alternative text information can be prepared as a candidate to replace the text information, and the alternative text information can be appropriately used as needed.

[0101] Regarding the dynamic image files including text information with low reliability, it is necessary to focus on correction on the basis of using them as teacher data, but by creating alternative text information, it is easy to find the dynamic image files of the correction object. In addition, regarding the text information with low reliability, it becomes easy to perform correction operations such as replacing it with alternative text information. The corrected dynamic image files can be used as teacher data for re-learning.

[0102] In addition, the judgment criteria (prescribed criteria) for whether to produce alternative text information can be set in advance to ensure a reasonable level of reliability as text information, or can be appropriately revised after being set. Furthermore, when producing alternative text information, the number of produced (i.e., the number of alternative candidates) is not particularly limited and can be determined arbitrarily.

[0103] Furthermore, as text information, when the first text information and the second text information are produced, Figure 5 As shown, as the related information, information related to the reliability of the second text information (hereinafter referred to as the second reliability information) can be prepared. The reliability of the second text information refers to an index indicating the accuracy (conversion accuracy) when the second text information is prepared by changing the language system. By preparing the second reliability information, when using the second text information, its reliability can be considered.

[0104] The reliability of the second text information is determined based on the consistency of the first text information corresponding to the second text information and the contents (specifically, types described later) of the plurality of sounds included in the sound data.

[0105] In one embodiment of the present invention, Figure 4 As shown, the incidental information of the sound also includes presence information and sound source information. If such information is included in the dynamic image file as incidental information, the usefulness of the dynamic image file is improved, for example, the accuracy of machine learning using the dynamic image file as teacher data can be improved.

[0106] The existence information is information related to whether the sound source of the sound included in the sound data exists within the field of view of the corresponding image frame. Figure 4As shown, the existence or non-existence information is information related to whether the speaker of the sound exists within the field of view of the image frame captured when speaking. Whether the sound source exists within the field of view can be determined based on the movement of the mouth of the sound source (i.e., the speaker within the field of view) recorded in the dynamic image data. Alternatively, in the case where the sound pickup microphone is a directional microphone, it can be determined whether the sound source within the field of view exists based on its sound pickup direction. Specifically, the sound pickup direction of the directional microphone can be set to face the space equivalent to the field of view, and when the direction of the sound deviates from its direction, it is determined that the sound source exists outside the field of view. In addition, the directional microphone is preferably a microphone that combines multiple microphone elements to collect sounds over a wide range of more than 180° (preferably 360°), and can determine the direction of each collected sound.

[0107] Sound source information is information related to the sound source, especially the speaker, such as Figure 4 As shown, each textualized sound, in other words, each text information is created and associated with the text information. In this way, by associating the sound source information with the text information, the speaker as the sound source and the text information of the sound can be associated with each other through the accompanying information (tag).

[0108] The sound source information may be, for example, identification information of a speaker serving as the sound source. The speaker identification information is information of the speaker determined based on the features of a region where the speaker exists in an image frame of the moving image data, and may be, for example, personal identification information such as the speaker's name or ID. As a method of identifying a speaker from a moving image or an image, a known subject recognition technology such as a face verification technology may be used.

[0109] In addition, as features of the region where the speaker exists in the image frame, the hue, saturation, brightness, shape, size, and position within the viewing angle of the region can be cited.

[0110] The sound source information may include information other than the above identification information, such as Figure 6 As shown, it includes location information, distance information, attribute information, etc.

[0111] The position information is information about the position of the sound source within the viewing angle, and more specifically, about the coordinate position of the sound source with the reference position within the viewing angle as the origin. There is no particular limitation on the method of determining the position, for example, Figure 7 As shown in FIG. 1 , in the field of view, a region surrounding a part or the whole of the sound source (hereinafter referred to as the sound source region) is defined. Then, when the sound source region is a rectangular region, two intersection points located at both ends of the diagonal line on the edge of the region (at Figure 7 The coordinates of the points indicated by white circles and black circles in FIG. 1 are determined as the positions (coordinate positions) of the sound sources. Figure 8 As shown in FIG. 1 , when the sound source region is a circular region, the position of the sound source can be determined based on the coordinates of the center (base point) of the region and the distance from the base point to the edge of the region (i.e., radius r). In addition, when the sound source region is a rectangle, the position of the sound source can also be determined based on the coordinates of the center (intersection of the diagonal lines) of the region and the distance from the center to the edge.

[0112] The distance information is information on the distance (depth) of the sound source within the viewing angle, and is, for example, a measurement result based on a distance measuring sensor mounted on the imaging device 20 .

[0113] The attribute information is information related to the attributes of the sound source within the viewing angle, specifically, information related to the attributes such as gender and age of the speaker within the viewing angle. The attributes of the speaker can be determined based on the characteristics of the region (i.e., the sound source region) where the speaker exists in the image frame of the dynamic image data, for example, by applying a known clustering method and determining the classification (class) to which the attributes of the speaker belong according to a predetermined classification standard.

[0114] In addition, the above-mentioned sound source information can be generated only for the sound source existing in the field of view angle, and can also be generated for the sound source outside the field of view angle. However, it is not limited to this. Even if it is a sound source (speaker) outside the field of view angle that is not recorded in the image frame, the voiceprint can be determined based on the voice (voice) of the speaker, and the recognition information of the speaker can be generated as the sound source information through the voiceprint verification technology.

[0115] In one embodiment of the present invention, Fig. 9 As shown, the relevant information may include, in addition to the reliability information and the second reliability information, mouth shape information, degree information, speaking method information, change information, language system information, type information, and error information.

[0116] like Fig.10 As shown, the mouth shape information is generated when the speaker, which is the sound source of the textualized sound, is present in the field of view, and is information related to the change in the shape of the speaker's mouth (i.e., the movement of the mouth) when the above sound is uttered. By recording the mouth shape information as incidental information in the dynamic image file, the dynamic image file can be more effectively used as teacher data for machine learning. Specifically, the dynamic image file including the mouth shape information is useful, for example, when implementing machine learning to construct a learning model for predicting speech sounds based on mouth movements.

[0117] In addition, the movement of the mouth can be identified from the image of the speaker recorded in the moving image data, more specifically, from the image of the mouth during speaking.

[0118] like Fig.10As shown, the degree information is information related to the degree of consistency between the speaker's mouth movement and the text information, and is produced when the mouth shape information is produced as related information. The degree of consistency is an indicator that represents the degree of consistency (match) between the speaker's mouth movement when uttering a speech sound and the text information of his voice. The degree of consistency can be said to be equivalent to one of the reliability of the text information, so by producing the degree information, the reliability of the text information can be determined from the speaker's mouth movement. That is, the degree information whose reliability is determined from the perspective of the degree of consistency between the speaker's mouth movement and the text information can be included in the dynamic image file. Therefore, when the dynamic image file is used as teacher data for implementing machine learning, the reliability of the text information can be further considered.

[0119] like Fig.11 As shown, the speaking method information is information related to the tone or intonation of the voice, and more specifically, information related to the tone or intonation when the text information is sent. Here, the concept of "tone" includes not only the strength of the voice in each word, but also the strength of the voice in a phrase, clause or sentence. Moreover, the concept of "tone" also includes the high and low pitch of the sound in a word, phrase or sentence. And "intonation" includes the ups and downs of a word, phrase, clause or sentence.

[0120] By creating the speaking method information as supplementary information, it is possible to utilize a dynamic image file including text information and speaking method information. Thus, for example, it is possible to construct a learning model (speech recognition model) that takes into account the correlation (correspondence) between the phrases, clauses or sentences shown in the text information and their speaking methods.

[0121] Furthermore, when the first text information and the second text information are created, utterance method information may be created as text information for each of the first text information and the second text information.

[0122] The change information and the language system information are both created when the first text information and the second text information are created as text information. In addition, as for the change information and the language system information, at least one of them can be created, and either one of them can be created, or both of them can be created.

[0123] like Figure 5 As shown, the change information is information related to the change of the language system (specifically, the change of the second text information to the language system). By creating the change information, it can be recognized that the second text information is created by changing the language system for the sound included in the sound data.

[0124] The language system information is information related to the language system of the first text information or the second text information, such as Figure 5The type of language system before and after the change is shown in FIG. The type of language system indicates classification such as Japanese, English, and Chinese, whether it is a dialect or a standard language, and in which region the dialect is spoken.

[0125] The change information and the language system information are both corresponding to the voice for which the second text information is created, such as Figure 5 As shown, the second text information and the first text information are associated with each other.

[0126] Type information is information related to the classification of sound content (hereinafter, also referred to as type). For example, in the case where the sound data includes the conversation sounds of multiple people, the type of the conversation sounds is determined by analyzing the sound data, such as Fig.12 As shown, type information about the determined type is generated.

[0127] The method of determining the type is not limited to the analysis of the sound data, and can also be determined based on the dynamic image data. Specifically, the image during the period when multiple sounds are generated in the dynamic image data (for example, during a conversation) can be analyzed to identify the scene or background of the image, and the type of the sound can be determined taking into account the identified scene or background. In this case, the scene or background of the image can be identified by using known subject detection technology and scene recognition technology.

[0128] In addition, the type is determined by an AI different from the AI ​​used to create the text information of the AI ​​for type determination.

[0129] For example, the type information is referred to when making the above-mentioned reliability information. That is, the textualized reliability information for a certain sound can be made according to the type of the sound. Specifically, when the content of the text information is consistent with the type of the sound, reliability information indicating a reliability higher than a prescribed benchmark can be made. On the contrary, when the content of the text information is inconsistent with the type of the sound, reliability information indicating a reliability lower than a prescribed benchmark can be made.

[0130] By creating type information, the content of the textualized sound (specifically, the meaning of the words in the text information) can be understood according to the type of the sound. For example, in a special type of conversation, when a certain word is used with a meaning unique to its type (a meaning different from the original meaning of the word), the meaning of the word can be correctly recognized.

[0131] Furthermore, by creating the type information and including the type information in the moving picture file as supplementary information, the video of the scene in which the sound of the type specified by the user is recorded can be found based on the type information. That is, the type information can be used as a search keyword when searching for moving picture files.

[0132] Error information is information related to the source of the sound, that is, the speaker's speaking error, specifically, Fig.13 The information shown is information indicating whether there is an error. A speech error refers to a slip of the tongue when uttering a sound (speech sound), a grammatical error, an error in the use of an auxiliary word, and a misuse of language. Whether there is a speech error is determined according to a predetermined standard, for example, by whether the following items are met.

[0133] Whether there are errors in the textualized voice (e.g., unnatural words, etc.)

[0134] ·Whether the usage (grammar) of the language in the textualized voice is correct

[0135] Whether the context is consistent with (matches with) the context determined based on the text information of each of the plurality of sounds included in the sound data

[0136] In addition, the presence or absence of speech errors is determined by an AI different from the error determination AI, specifically, the AI ​​used for the production of text information. Furthermore, for a speech with a speech error, a speech sound with the error corrected (hereinafter referred to as a corrected sound) can be predicted, and text information of the corrected sound can be further produced.

[0137] By creating error information and using the dynamic image file including the error information as teacher data to implement machine learning, the accuracy of its learning can be improved. Specifically, when weights are set for dynamic image files used as teacher data and machine learning is implemented by reflecting the weights, the weights of files containing speech errors in the voices included in the voice data are reduced. Through this weighting, more appropriate learning results can be obtained in machine learning.

[0138] In one embodiment of the present invention, Fig. 9 As shown, the accompanying information of the sound may further include link destination information and rights relationship information.

[0139] The link destination information is information indicating a link to a storage destination (save destination) of the voice file when the same voice data as the voice data of the moving image file is created as a separate file (voice file). In addition, the voice data of the moving image file includes a plurality of voices from a plurality of sound sources (speakers), and the voice file may be created for each sound source (each speaker). In this case, the link destination information is created for each voice file (that is, for each speaker).

[0140] The rights relationship information is information about the ownership of the rights related to the sounds included in the sound data and the ownership of the rights related to the moving image data. For example, when a moving image file is produced by shooting scenes of multiple artists singing songs in sequence, the rights (copyright) of the moving image data belong to the producer of the moving image file (that is, the photographer of the image). On the other hand, the rights related to the respective sounds (singing) of the multiple artists recorded in the sound data belong to each artist or the group to which they belong. In this case, the rights relationship information stipulating the ownership relationship of these rights is produced.

[0141] <<About the functions of the information production device>>

[0142] refer to Fig.14 , the functions of the information production device 10 involved in one embodiment of the present invention are explained.

[0143] like Fig.14 As shown, the information production device 10 includes an acquisition unit 21, a determination unit 22, a first production unit 23, a second production unit 24, a statistical processing unit 25, a display unit 26, an analysis unit 27, and a notification unit 28. These functional units are implemented by the cooperation of the hardware devices (processor 11, memory 12, communication interface 13, input device 14, output device 15, and storage device 16) of the information production device 10 and the software including the above-mentioned information production program. In addition, artificial intelligence (AI) is used to implement some functions.

[0144] Each functional unit will be described below.

[0145] (Acquisition Department)

[0146] The acquisition unit 21 controls each unit of the camera device 20 to acquire dynamic image data and sound data. In one embodiment of the present invention, when multiple sound sources emit sounds (speech sounds) in sequence, the acquisition unit 21 synchronizes the dynamic image data and the sound data and produces these data at the same time. Specifically, the acquisition unit 21 acquires dynamic image data composed of multiple image frames in a manner that records at least one sound source in one image frame. In addition, the acquisition unit 21 acquires sound data including multiple sounds from multiple sound sources recorded in multiple image frames included in the dynamic image data. At this time, each sound corresponds to two or more image frames acquired (photographed) during the period when the sound is generated in the multiple image frames.

[0147] (Confirmation Department)

[0148] The identification unit 22 identifies the content related to the sound included in the sound data based on the moving image data and the sound data acquired by the acquisition unit 21 .

[0149] Specifically, the identification unit 22 identifies the correspondence between each of the plurality of sounds included in the sound data and the image frames, and identifies two or more image frames acquired during a period in which the sound is generated.

[0150] Then, the identification unit 22 identifies the sound source (speaker) of each sound.

[0151] Furthermore, the determination unit 22 determines whether the sound source of the sound exists within the field of view of the corresponding image frame. Furthermore, when the sound source exists within the field of view, the determination unit 22 determines the position and distance (depth) of the sound source within the field of view, and determines the attribute and identification information of the sound source. Furthermore, the determination unit 22 determines the movement of the mouth of the sound source (speaker) existing within the field of view during speaking.

[0152] Furthermore, the identification unit 22 identifies the types of the plurality of sounds included in the sound data (specifically, the classification of the conversation contents, etc.).

[0153] Furthermore, the identification unit 22 identifies the speaking method such as the intonation of each voice.

[0154] Then, the identification unit 22 identifies the presence or absence of a utterance error and the content of the utterance error for each voice.

[0155] (1st Production Department and 2nd Production Department)

[0156] The first creating unit 23 and the second creating unit 24 each create supplementary information of the sound for each of the plurality of sounds included in the sound data.

[0157] The first production unit 23 produces text information by converting the sound into text. In one embodiment of the present invention, the first production unit 23 maintains the language system of the sound and converts the sound into text to produce text information (more specifically, first text information). In addition, the first production unit 23 can produce second text information by changing the language system and converting the sound into text.

[0158] The second creating unit 24 creates information other than text information (hereinafter also referred to as non-text information) in the incidental information of the sound.

[0159] Specifically, the second creating unit 24 creates correspondence information on the correspondence relationship between the sound and the image frame, based on the correspondence relationship between the sound and the image frame determined by the determining unit 22 .

[0160] Furthermore, the second production unit 24 produces relevant information related to the textualization of the sound. The relevant information includes reliability information related to the reliability of the textualization of the sound. In this case, the second production unit 24 can produce the reliability information based on the type of the sound determined by the determination unit 22. In detail, the second production unit 24 can produce the reliability information based on the consistency between the type of the sound and the content of the text information.

[0161] Furthermore, when the second text information is produced as the text information, the second producing unit 24 produces second reliability information related to the reliability of the second text information as the relevant information. In addition, the second producing unit 24 produces at least one of change information related to the change of the second text information to the language system and language system information related to the language system of the first text information or the second text information as the relevant information.

[0162] Furthermore, the second creating unit 24 creates, as the related information, utterance method information related to the utterance method based on the utterance method of the voice identified by the identifying unit 22 .

[0163] Furthermore, based on the type of the sound determined by the determination unit 22, the second creation unit 24 creates type information related to the type as related information.

[0164] Furthermore, the second creation unit 24 creates mouth shape information related to the mouth movement of the sound source (speaker) as the relevant information based on the mouth movement of the sound source (speaker) determined by the determination unit 22. In this case, the second creation unit 24 may further create degree information related to the degree of consistency between the mouth movement of the speaker and the text information as the relevant information.

[0165] Furthermore, when the speaker's utterance error is identified by the identification unit 22, the second creation unit 24 creates error information related to the utterance error as the related information.

[0166] Furthermore, the second creating unit 24 creates presence / absence information related to whether the sound source of the sound exists within the field of view of the corresponding image frame as information other than the relevant information. Furthermore, the second creating unit 24 creates sound source information related to the sound source existing within the field of view, specifically, identification information, position information, distance information, attribute information, etc. of the sound source.

[0167] Then, regarding text information (strictly speaking, first text information) whose reliability is lower than a predetermined standard, the second creating unit 24 creates alternative text information related to a text different from the text information as an alternative candidate.

[0168] In addition, the second creating unit 24 only needs to create at least the reliability information among the above-mentioned non-text information, and may omit the creation of other non-text information.

[0169] (Statistical Processing Department)

[0170] The statistical processing unit 25 performs statistical processing on the incidental information of each of the multiple sounds included in the sound data, that is, the sounds from multiple sound sources, to obtain statistical data, which is data indicating a statistical quantity related to the reliability of the text information generated for each sound.

[0171] Specifically, the statistical processing unit 25 performs statistical processing on the text information of each sound and the reliability information prepared for each text information. Fig.15 The reliability distribution (for example, frequency distribution) of the text information is shown. This makes it easy to identify the image frame of the sound with low reliability or the sound with high reliability when the text is converted, and to analyze the sound that is the cause.

[0172] In addition, the statistical processing may be performed, for example, on all the moving image files produced in the past, and the incidental information of the sounds included in each moving image file may be collected as a whole and statistical processing may be performed. Alternatively, the incidental information of the sounds included in the moving image files specified by the user may be collected as a whole and statistical processing may be performed. Furthermore, in one moving image file, the incidental information of the sounds corresponding to the moving images of the period specified by the user may be collected as a whole and statistical processing may be performed.

[0173] (Display)

[0174] The display unit 26 displays the statistical data obtained by the statistical processing unit 25 (for example, Fig.15 The screen displaying the statistical data may be formed by the display of the camera 20 or by an external display of another device connected to the camera 20.

[0175] As described above, by displaying statistical data related to the reliability of text information, the user can visually grasp the accuracy of the reliability of the text information, its tendency, and the like.

[0176] (Analysis Department)

[0177] When the reliability indicated by the reliability information on text information of a certain sound falls below a predetermined standard, the analyzing unit 27 analyzes the cause thereof.

[0178] Specifically, the analysis unit 27 reads a moving image file produced in the past. The analysis unit 27 determines the reason why the reliability is lower than a predetermined reference based on the text information, reliability information, and non-text information other than the text information in the accompanying information of the sound included in the read moving image file. Here, the non-text information includes, for example, existence information, sound source information, correspondence information, change information, and language system information.

[0179] More specifically, when using the presence or absence information as non-text information, the analysis unit 27 determines the correlation between the presence or absence of the sound source (speaker) within the viewing angle and the reliability of the text information of the sound (speech sound) emitted by the sound source. Then, the analysis unit 27 determines the reason why the reliability is lower than the prescribed reference by establishing a corresponding association with the presence or absence of the sound source within the viewing angle based on the determined correlation.

[0180] Furthermore, when sound source information (e.g., identification information of a speaker as a sound source) is used as non-text information, the analysis unit 27 determines the correlation between the identification information of the speaker and the reliability of the text information of the speaker's speech sound. Then, based on the determined correlation, the analysis unit 27 establishes a corresponding association with the identification information of the speaker and determines the reason why the reliability becomes lower than a predetermined reference. Furthermore, when identification information of a sound source other than speech sound (e.g., wind sound or car driving sound) is available as non-text information, the reason why the reliability becomes lower than a predetermined reference is determined by establishing a corresponding association with the identification information of the sound source and determining the reason why the reliability becomes lower than a predetermined reference.

[0181] Furthermore, when the corresponding information is used as non-text information, the analysis unit 27 determines the period of time during which the text-converted sound is generated (in other words, the length of the text), and determines the correlation between the length of the text and the reliability of the text information. Then, based on the determined correlation, the analysis unit 27 determines the reason why the reliability becomes lower than a predetermined reference by establishing a correspondence relationship with the length of the text.

[0182] Furthermore, when the change information or language system information is used as non-text information, the analysis unit 27 determines the language system of the text information based on the change information / language system information. For example, the analysis unit 27 determines whether the language system of the text information is a standard language or a dialect, and in the case of a dialect, determines which region the dialect is. Then, the analysis unit 27 determines the relationship between the language system of the text information and the reliability. Then, based on the determined correlation, the analysis unit 27 establishes a corresponding association between the reason when the reliability is lower than the prescribed benchmark and the language system of the text information to determine.

[0183] (Notification Department)

[0184] The notification unit 28 notifies the user of the reason for the target sound identified by the analysis unit 27, that is, the reason why the reliability of the sound with respect to the text is lower than a predetermined reference. Thus, the user can easily understand the reason why the sound has low reliability with respect to the text information.

[0185] In addition, there is no particular limitation on the mechanism for notifying the cause, and for example, text information related to the cause may be displayed on the screen, or voice related to the cause may be output.

[0186] <<About the information creation process according to one embodiment of the present invention>>

[0187] Next, the information production flow using the information production device 10 is described. In the information production flow described below, the information production method of the present invention is used. That is, each step in the information production flow described below corresponds to the constituent elements of the information production method of the present invention.

[0188] The following process is merely an example, and unnecessary steps in the process may be deleted, new steps may be added to the process, or the execution order of two steps in the process may be replaced without departing from the spirit of the present invention.

[0189] Each step (process) in the information production flow is executed by the processor 11 included in the information production device 10. That is, in each process in the information production flow, the processor 11 executes processing corresponding to each process in data processing specified by the information production program.

[0190] In one embodiment of the present invention, the information production process is divided into: Fig.16 The main processes and Fig.18 The following is a description of each process.

[0191] (Main Process)

[0192] In the main process, the moving image data and the sound data are acquired, and the accompanying information of the moving image data is created to create a moving image file.

[0193] In the main flow, the processor 11 performs a first acquisition step ( S001 ) of acquiring sound data including a plurality of sounds from a plurality of sound sources and a second acquisition step ( S002 ) of acquiring moving image data including a plurality of image frames.

[0194] In addition, Fig.16 In the illustrated flow, the second acquisition step is performed after the first acquisition step. However, for example, when a moving image with sound is captured using the imaging device 20 , the first acquisition step and the second acquisition step are performed simultaneously.

[0195] During the execution of the first acquisition step and the second acquisition step, the processor 11 executes a determination step (S003) and a production step (S004). In the determination step, the content related to the sound included in the sound data is determined, specifically, the correspondence between the sound and the image frame, the speaking method of the sound, the presence or absence of speech errors, and the content of the speech errors are determined.

[0196] Furthermore, in the determination process, it is determined whether the sound source of the sound exists within the field of view of the corresponding image frame. If the sound source exists within the field of view, the position and distance of the sound source within the field of view, the attributes of the sound source and identification information are further determined.

[0197] Then, in the specifying step, the movement of the mouth of the sound source (speaker) during speaking that is present within the viewing angle is specified.

[0198] Furthermore, in the determining step, types of a plurality of sounds included in the sound data are determined.

[0199] Production process according to Fig.17 The process shown in the figure is performed. In the production process, as the accompanying information of the dynamic image data, the accompanying information of the sound is produced. Specifically, the step (S011) of producing text information by texting the sound included in the sound data is implemented. In this step S011, the language system of the sound is maintained and the sound is texted to produce text information (strictly speaking, the first text information), and in detail, text information related to the phrase, clause or sentence texted with the sound is produced.

[0200] Furthermore, when the language system is changed to create text information (second text information), the second text information is created together with the first text information (S012, S013).

[0201] When the second text information is created, change information related to the change of the language system or language system information related to the language system of the first text information or the second text information may be created together. Furthermore, second reliability information related to the reliability of the second text information may be created.

[0202] Furthermore, in the production process, a step (S014) is implemented to produce the relevant information, i.e., reliability information, for the sound that produced the text information. In this step S014, the reliability of the text is determined using an algorithm or a learning model that calculates the accuracy of the phrase, clause, or sentence that has been converted into text, and reliability information related to the reliability is produced. Furthermore, the reliability information can also be produced based on the clarity of the sound (speech sound) and the presence or absence of noise.

[0203] Furthermore, when creating the reliability information in the above-mentioned manner, the contents determined in the determination step S003 (specifically, the type of sound and the movement of the speaker's mouth, etc.) can be referred to and the reliability information can be created based on them.

[0204] Furthermore, when there is text information (strictly speaking, the first text information) whose reliability indicated by the reliability information is lower than a predetermined standard, alternative text information of a text different from the above text information is generated as an alternative candidate (S015, S016).

[0205] Furthermore, in the production process, existence information related to whether the sound source of the sound that produced the text information exists within the field of view of the corresponding image frame is produced (S017). Furthermore, if the sound source exists within the field of view, sound source information related to the sound source is produced, specifically, position information, distance information, identification information, attribute information, etc. of the sound source within the field of view are produced (S018, S019).

[0206] In the production process, other related information (specifically, corresponding information, mouth shape information, degree information, type information, error information, speaking method information, etc.) is further produced (S020).

[0207] During the acquisition of the moving image data and the sound data (i.e., during the moving image shooting), the determination process and the production process are repeatedly performed. Then, when the acquisition of these data is completed (S005), the determination process and the production process are completed accordingly, and the main flow ends.

[0208] As a result of executing the above series of processes in the main process, the supplementary information of the sound including text information and reliability information is produced for each of the multiple sounds included in the sound data. Then, when the main process is completed, the supplementary information is added to the dynamic image data and the sound data, and a dynamic image file including the dynamic image data, the sound data and the supplementary information is produced.

[0209] (Sub-process)

[0210] The sub-process is implemented separately from the main process, for example, after the main process ends. In the sub-process, first, a step (S031) of performing statistical processing on the sound data of the target moving image file is implemented. In this step S031, statistical processing is performed on the reliability information generated for each of the multiple sounds included in the sound data to determine the distribution of the reliability (refer to Fig.15 Then, a display step is performed, in which statistical data obtained by statistical processing, that is, data indicating distribution of reliability, is displayed (S032).

[0211] Furthermore, in the sub-process, when there is text information whose reliability is lower than a predetermined standard among the text information of the target dynamic image file, an analysis step and a notification step (S033, S034) are implemented. In the analysis step, the reason why the reliability of the text information is lower than the predetermined standard is determined based on non-text information other than the text information. More specifically, the correlation between the reliability of the text information and the content determined from the non-text information is determined, and the above-mentioned reason is determined (estimated) based on the correlation.

[0212] In the notification step, the cause identified in the analysis step is notified to the user, thereby enabling the user to understand the cause of the text information having reliability lower than a predetermined standard.

[0213] Then, when the above steps are completed, the sub-process ends.

[0214] <<Other implementation methods>>

[0215] The above-described embodiment is a specific example given to explain the information production method, information production device, and moving image file of the present invention in an easily understandable manner, but is merely an example and other embodiments are also conceivable.

[0216] (Regarding files recording moving image data and sound data)

[0217] In the above embodiment, a video camera 20 with a microphone is used to capture a moving image with sound, and the moving image data and the sound data are simultaneously acquired, and these data are included in a moving image file. However, this is not limited to this. The moving image data and the sound data may also be acquired by other devices, and each data may be recorded in another file. In this case, it is preferred to synchronize and acquire each of the moving image data and the sound data.

[0218] (Regarding the structure of the information production device)

[0219] In the above-mentioned embodiment, the structure in which the information production device of the present invention is mounted on the camera device is described. That is, in the above-mentioned embodiment, the accompanying information of the dynamic image data is produced by the camera device that obtains both the dynamic image data and the sound data. However, it is not limited to this, and the accompanying information can also be produced by a device different from the camera device, specifically, by a PC, a smart phone, or a tablet terminal connected to the camera device. That is, the information production device can also be composed of a computer of a device different from the camera device, obtain the dynamic image data and the sound data from the camera device, and produce the accompanying information of the dynamic image data (specifically, the accompanying information of the sound).

[0220] (About speaker list)

[0221] In the above embodiment, for each of the plurality of sounds (speech sounds) included in the sound data, identification information of the speaker as the sound source is prepared as the supplementary information (sound source information). Fig.19 The speaker list is shown. The speaker list is created by listing the speakers that are the sound sources in time series for each of the multiple sounds included in the sound data, and associating them with the moving image file including the sound data. In the speaker list, the speaker of each speech sound and the image frame corresponding to the speech sound are defined by associating them with the image frame at the start time of the sound generation (start frame) and the image frame at the end time of the sound generation (end frame).

[0222] (Regarding changes in the processes involved in information production)

[0223] As described above, the information production process of the present invention is not limited to the process involved in the above-mentioned embodiment, and may also include Figures 16 to 18 For example, in the information production process, a determination step may be further performed to determine whether there is a change in the audio data or the dynamic image data in the dynamic image file.

[0224] In the determination process, the presence or absence of a change (tampering) is determined based on the text information in the accompanying information of the sound included in the dynamic image file and the mouth shape information corresponding to the text information. Specifically, in the determination process, the processor 11 determines whether the content of the text information is consistent with the mouth movement shown by the corresponding mouth shape information. The corresponding mouth shape information is information related to the mouth movement of the speaker of the sound determined from the dynamic image during the generation of the sound (speech sound) texted in the dynamic image data. Then, when the two are inconsistent, the processor 11 determines that "there is a change".

[0225] By implementing the above determination process, when there is a change (tampering) in the sound data or the dynamic image data, it is possible to identify the situation. Thus, when using a dynamic image file, the credibility of the sound data and dynamic image data included in the dynamic image file (whether it is tampered data) can be confirmed.

[0226] (About the structure of the processor)

[0227] The processor 11 included in the information production device of the present invention includes various processors. Among the various processors, for example, a general-purpose processor, namely, a CPU, which executes software (program) to function as various processing units is included.

[0228] In addition, various processors include PLDs (Programmable Logic Devices) which are processors whose circuit configuration can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays).

[0229] Furthermore, various processors include processors such as ASIC (Application Specific Integrated Circuit) which have a circuit configuration specifically designed to execute specific processing, that is, dedicated circuits.

[0230] Furthermore, one of the various processors described above may constitute one functional unit of the information production device of the present invention. Alternatively, one functional unit of the information production device of the present invention may be constituted by a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs or a combination of FPGAs and CPUs.

[0231] Furthermore, the plurality of functional units of the information production device of the present invention may be constituted by one of various processors, or two or more of the plurality of functional units may be constituted by one processor.

[0232] Furthermore, as in the above-described embodiment, a combination of one or more CPUs and software may constitute one processor, and the processor may function as a plurality of functional units.

[0233] Furthermore, the following method may be adopted: for example, a processor that realizes the functions of the entire system including multiple functional units in the information production device of the present invention by using one IC (Integrated Circuit) chip, as represented by SoC (System on Chip). Furthermore, the hardware structure of the above-mentioned various processors may be a circuit (Circuitry) composed of circuit elements such as semiconductor elements.

[0234] Explanation of symbols

[0235] 10-information production device, 11-processor, 12-memory, 13-communication interface, 14-input device, 15-output device, 16-storage device, 20-camera device, 21-acquisition unit, 22-determination unit, 23-first production unit, 24-second production unit, 25-statistical processing unit, 26-display unit, 27-analysis unit, 28-notification unit.

Claims

1. A method for producing information, comprising: A first acquisition step of acquiring sound data including a plurality of sounds from a plurality of sound sources; and A production step of producing text information for converting the sound into text and related information related to the text conversion of the sound as supplementary information of the dynamic image data corresponding to the sound data.

2. The information production method according to claim 1, wherein: The relevant information includes reliability information related to the reliability of the textualization of the sound.

3. The information production method according to claim 2, comprising: The second acquisition step is to acquire the dynamic image data including a plurality of image frames. In the production step, correspondence information indicating a correspondence relationship between two or more image frames among the plurality of image frames and the text information is produced as the supplementary information, The text information is information related to a phrase, a clause or a sentence that converts the sound into text. The reliability information is information related to the reliability of the phrase, clause, or sentence with respect to the sound.

4. The information production method according to claim 1, comprising: The second acquisition step is to acquire the dynamic image data including a plurality of image frames. In the creation step, sound source information related to the sound source and presence / absence information related to whether the sound source exists within a field of view of the corresponding image frame are created as the supplementary information.

5. The information production method according to claim 2, wherein: When the reliability of the text information is lower than a predetermined reference, in the creating step, substitute text information related to a text different from the text information is created for the sound.

6. The information production method according to claim 1, wherein: The related information includes error information related to a speaking error of a speaker serving as the sound source.

7. The information production method according to claim 2, wherein: In the creation step, the reliability information is created based on the classification of the content of the sound.

8. The information production method according to claim 1, wherein: In the creating step, sound source information related to a speaker as the sound source is created. The related information includes degree information related to the degree of consistency between the speaker's mouth movement and the text information.

9. The information production method according to claim 1, wherein: The related information includes speaking method information related to a speaking method of the sound.

10. The information production method according to claim 1, wherein: In the production process, first text information and second text information are produced as the text information, the first text information maintains the language system of the sound and converts the sound into text, and the second text information changes the language system and converts the sound into text, The related information includes language system information related to the language system of the first text information or the second text information or change information related to a change of the second text information to the language system.

11. The information production method according to claim 10, wherein: The related information includes information related to the reliability of the second text information.

12. The information production method according to claim 2, wherein: In the production step, the text information and the reliability information are produced for each of the plurality of sounds. The information generating method further includes a display step of displaying statistical data obtained by statistically processing the reliability information generated for each of the plurality of sounds.

13. The information production method according to claim 2, further comprising: an analyzing step of analyzing a reason why the reliability has fallen below a predetermined reference when the reliability indicated by the reliability information has fallen below a predetermined reference; and The process is notified of the reasons stated.

14. The information production method according to claim 13, wherein: When the information other than the text information in the supplementary information is set as non-text information, in the analyzing step, the cause is identified based on the non-text information.

15. The information production method according to claim 1, further comprising: The determination step is to determine whether or not there is a change in the audio data or the dynamic image data based on the movement of a mouth of a speaker of the audio in the text information and the dynamic image data.

16. The information production method according to claim 1, wherein: The sound is speech sound.

17. An information production device comprising a processor, The processor performs the following processing: Acquiring sound data including a plurality of sounds from a plurality of sound sources; and Text information for converting the sound into text and related information related to the text conversion of the sound are created as supplementary information of the moving image data corresponding to the sound data.

18. A dynamic image file, comprising: sound data including a plurality of sounds from a plurality of sound sources; dynamic image data corresponding to the sound data; and The accompanying information of the dynamic image data, The supplementary information includes text information for converting the sound into text and related information related to the text conversion of the sound.

Images