Data creation method and recording medium

Abstract

The present invention provides a data creation method and a recording medium, which can effectively adjust the volume of the audio of each subject according to the state of each subject of multiple subjects. The data creation method includes the following steps: an association establishment step (step S10 and step S11), establishing an association between a microphone and each subject having each microphone; a video recording step (step S12, step S20, step S21), using a camera device to record dynamic image data; a recording step (step S13), using each microphone to record the audio data of each subject in synchronization with the start of the video recording step; a detection step (step S14, step S15), automatically detecting the state of the subject during the video recording step; and an assignment step (step S16 to step S19), according to the result of the detection step, assigning an identification code related to the volume adjustment of the audio data of each subject to the dynamic image data.

Description

Technical Field

[0001] The present invention relates to a data creation method and a recording medium. Background Art

[0002] Conventionally, there is a technology that collects audio from an imaging device that captures moving image data using a microphone connected wirelessly, for example, to obtain audio data synchronized with the moving image data.

[0003] Patent Document 1 describes a technology for storing an audio signal in a recording medium when a wireless microphone connected to an imaging device cannot transmit an audio signal to the imaging device.

[0004] Patent Document 2 describes a wireless communication device that generates a recording file by associating log information related to communication between a microphone and the wireless communication device.

[0005] Previous technical literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2015-73170

[0008] Patent Document 2: Japanese Patent Application Publication No. 2015-119229 Summary of the invention

[0009] One embodiment of the technology according to the present invention provides a data creation method that can efficiently adjust the volume of audio of each subject according to the state of each of a plurality of subjects.

[0010] Means for solving technical problems

[0011] A data creation method as one aspect of the present invention is used in a camera system having a camera device and a plurality of microphones connected to the camera device, the data creation method comprising the following steps: an association establishing step of establishing an association between the microphones and the subjects having the microphones; a video recording step of recording dynamic image data using the camera device; an audio recording step of recording audio data of the subjects using the microphones in synchronization with the start of the video recording step; a detection step of automatically detecting the state of the subject during the video recording step; and an assignment step of assigning an identification code related to the volume adjustment of the audio data of each subject to the dynamic image data based on the result of the detection step.

[0012] Preferably, in the detection step, the state in which the subject is making a sound is identified in the dynamic image data by image processing, and in the assignment step, the dynamic image data is assigned an identification code that increases the volume of the audio data of the subject making the sound relative to other audio data.

[0013] Preferably, in the detection step, the direction in which each subject is facing is identified in the dynamic image data by image processing, and in the assignment step, an identification code for adjusting the volume of audio data according to the direction of the face of each subject relative to the camera device is assigned to the dynamic image data.

[0014] Preferably, in the detecting step, the distance between each subject and the imaging device is identified in the moving image data by image processing, and in the assigning step, an identification code for adjusting the volume of the audio data according to the distance of each subject is assigned to the moving image data.

[0015] Preferably, in the detection step, image processing is used to identify whether each subject exists within the field of view of the camera device in the dynamic image data, and in the assignment step, an identification code is assigned to the dynamic image data for adjusting the volume of the audio data according to whether the subject exists within the field of view of the camera device.

[0016] Preferably, the data creation method includes a position acquisition step, obtaining the position of each of the multiple subjects from a position detection system possessed by each of the multiple subjects, a detection step detecting the position of each of the multiple subjects obtained by the position acquisition step, and in an assignment step, assigning an identification code related to the volume adjustment of the audio data of each subject to the dynamic image data based on the result of the detection step.

[0017] Preferably, the data creation method includes, after the providing step, an accepting step of accepting volume adjustment of the audio data by a user.

[0018] As another embodiment of the present invention, a data creation method is used in a camera system having a camera device and multiple microphones connected to the camera device, and the data creation method includes the following steps: an association establishment step of establishing an association between the microphones and the subjects having the microphones; a video recording step of recording dynamic image data using the camera device; a sound recording step of recording audio data of each subject using each microphone in synchronization with the start of the video recording step; a detection step of automatically detecting the state of the subject during the video recording step; a synthesis step of synthesizing the audio data and the dynamic image data; and an adjustment step of automatically adjusting the volume of the audio data of each subject before or after the synthesis step based on the result of the detection step.

[0019] Preferably, in the detecting step, a state in which the subject is making a sound is recognized in the moving image data by image processing, and in the adjusting step, the volume of the audio data of the subject making the sound is increased relative to other audio data.

[0020] Preferably, in the detecting step, the direction in which each subject is facing is recognized by image processing in the moving image data, and in the adjusting step, the volume of the audio data is adjusted according to the direction of the face of each subject relative to the imaging device.

[0021] Preferably, in the detecting step, the distance between each subject and the imaging device is recognized in the moving image data by image processing, and in the adjusting step, the volume of the audio data is adjusted according to the distance of each subject.

[0022] Preferably, in the detection step, whether each subject exists within the field of view of the camera device is identified by image processing in the dynamic image data, and in the adjustment step, the volume of the audio data is adjusted according to whether the subject exists within the field of view of the camera device.

[0023] Preferably, the data creation method includes a position acquisition process, obtaining the position of each of the multiple subjects from a position detection system possessed by each of the multiple subjects, and in an adjustment process, adjusting the volume of the audio data of each subject based on the results of the position acquisition process and the results of the detection process.

[0024] As another embodiment of the present invention, a data creation program is used in a camera system having a camera device and multiple microphones connected to the camera device, and the data creation program includes the following steps: an association establishment step of establishing an association between the microphones and each subject having each microphone; a video recording step of recording dynamic image data using the camera device; an audio recording step of recording audio data of each subject using each microphone in synchronization with the start of the video recording step; a detection step of automatically detecting the state of the subject from the dynamic image data during the video recording step; and an assignment step of assigning an identification code related to the volume adjustment of the audio data of each subject to the dynamic image data based on the result of the detection step.

[0025] As another embodiment of the present invention, a data creation program is used in a camera system having a camera device and multiple microphones connected to the camera device, and the data creation program includes the following steps: an association establishment step for establishing an association between the microphones and the subjects having the microphones; a video recording step for recording dynamic image data using the camera device; a sound recording step for recording audio data of each subject using each microphone in synchronization with the start of the video recording step; a detection step for automatically detecting the state of the subject from the dynamic image data during the video recording step; a synthesis step for synthesizing the audio data and the dynamic image data; and an adjustment step for automatically adjusting the volume of the audio data of each subject before or after the synthesis step based on the result of the detection step. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a diagram conceptually showing a camera system.

[0027] Figure 2 This is a block diagram showing the schematic structure of a camera system.

[0028] Figure 3 This is a block diagram of the main functions implemented by the CPU when recording moving image data and audio data.

[0029] Figure 4 is a flowchart illustrating the data creation method.

[0030] Figure 5 This is a diagram for explaining an example of moving image data to which an identification code is assigned.

[0031] Figure 6 This is a block diagram of the main functions implemented by the CPU when recording moving image data and audio data.

[0032] Figure 7 is a flowchart illustrating the data creation method.

[0033] Figure 8 This is a diagram for explaining volume adjustment of the first audio data and the second audio data.

[0034] Fig. 9 This is a block diagram showing the schematic structure of a camera system.

[0035] Fig.10 This is a block diagram of the main functions implemented by the CPU when recording moving image data and audio data.

[0036] Fig.11 It is a figure explaining a specific example.

[0037] Fig.12 It is a figure explaining a specific example.

[0038] Fig.13 It is a figure explaining a specific example.

[0039] Fig.14 It is a figure explaining a specific example. DETAILED DESCRIPTION

[0040] Hereinafter, preferred embodiments of the data creation method and the data creation program according to the present invention will be described with reference to the accompanying drawings.

[0041] Figure 1 This is a diagram conceptually showing a camera system using the data creation method of the present invention.

[0042] The imaging device 1 constituting the camera system 100 acquires moving image data by capturing a video of a person A and a person B. The person A has a first microphone 12, and the person B has a second microphone 14. The first microphone 12 and the second microphone 14 are wirelessly connected to the imaging device 1.

[0043] In the following description, an example using two microphones (the first microphone 12 and the second microphone 14) is described, but the number of microphones is not particularly limited, and the camera system 100 can use multiple microphones. Furthermore, the first microphone 12 and the second microphone are connected to the imaging device 1 wirelessly, but may also be connected by wire.

[0044] Figure 2 : is a block diagram showing a schematic configuration of the camera system 100 .

[0045] The camera system 100 includes an imaging device 1 , a first microphone 12 , and a second microphone 14 .

[0046] The camera device 1 includes an image pickup unit 10, a display unit 16, a storage unit 18, an audio output unit 20, an operation unit 22, a CPU (Central Processing Unit) 24, a ROM (Read Only Memory) 26, a RAM (Random Access Memory) 28, and a third wireless communication unit 30. In addition, the camera device 1 is wirelessly connected to the first microphone 12 via the first wireless communication unit 12B and the third wireless communication unit 30, and is wirelessly connected to the second microphone 14 via the second wireless communication unit 14B and the third wireless communication unit 30.

[0047] The imaging unit 10 captures video to obtain dynamic image data. The imaging unit 10 includes an imaging optical system 10A, an imaging element 10B, and an image signal processing unit 10C. The imaging optical system 10A forms an image of a subject on a light receiving surface of the imaging element 10B. The imaging element 10B converts the image of the subject formed on its light receiving surface by the imaging optical system 10A into an electrical signal. The image signal processing unit 10C performs a predetermined signal processing on the signal output from the imaging element 10B to generate dynamic image data.

[0048] The first microphone 12 collects the audio (first audio) of person A. The first microphone 12 includes a first audio signal processing unit 12A and a first wireless communication unit 12B. The first audio signal processing unit 12A performs a predetermined signal processing on the signal from the microphone to generate the first audio data of the first audio. The first wireless communication unit 12B converts the first audio data into a wireless signal according to the communication method specified in the specification of Bluetooth (a registered trademark), performs the processing required for wireless communication, and wirelessly outputs it to the camera device 1. In addition, the wireless communication method is not particularly limited to Bluetooth, and other methods can also be used. For example, DECT (Digital Enhanced Cordless Telecommunication), wireless LAN (Local Area Network) or Zigbee (a registered trademark) can be used as the wireless communication method. In addition, the second microphone 14 has the same structure as the above-mentioned first microphone 12, so the description is omitted.

[0049] The display unit 16 displays the video corresponding to the dynamic image data acquired by the camera unit 10 in real time. In addition, the display unit 16 displays the video to be played. In addition, the display unit 16 displays an operation screen, a menu screen, a message, etc. as needed. The display unit 16 includes, for example, a display device such as an LCD (Liquid Crystal Display) and its drive circuit, etc.

[0050] The storage unit 18 mainly records the acquired moving image data and audio data. The storage unit 18 is configured to include, for example, a storage medium such as a nonvolatile memory and a control circuit thereof.

[0051] The audio output unit 20 outputs audio played according to audio data. In addition, the audio output unit 20 outputs a warning sound, etc. as needed. The audio output unit 20 includes a speaker and a data processing circuit that processes audio data of the audio output from the speaker.

[0052] The operation unit 22 receives operation input from the user and is configured to include various operation buttons such as a recording button, buttons displayed on the display unit 16 and a detection circuit for their operation, and the like.

[0053] The CPU 24 functions as the control unit of the entire device by executing a predetermined control program. The CPU 24 controls the operation of each unit according to the user's operation and centrally controls the operation of the entire device. The ROM 26 records various programs executed by the CPU 24 and data required for control. The RAM 28 provides a working storage space for the CPU 24.

[0054] The third wireless communication unit 30 receives wireless signals output from the first wireless communication unit 12B and the second wireless communication unit 14B, and processes the received wireless signals according to the Bluetooth standard. The imaging device 1 obtains the first audio data and the second audio data via the third wireless communication unit 30 .

[0055] <First embodiment>

[0056] The first embodiment of the present invention is described. In this embodiment, an identification code related to the volume adjustment of audio data is assigned to the dynamic image data according to the state of the subject automatically detected from the dynamic image data. Therefore, in this embodiment, in the editing work performed after acquiring the dynamic image data, the user can adjust the volume according to the identification code, thereby saving the trouble of confirming the images one by one and effectively adjusting the volume of the audio data.

[0057] Figure 3 This is a block diagram of the main functions implemented by the CPU when recording dynamic image data and audio data. Figure 3 As shown, the CPU 24 functions as an imaging control unit 101 , an image processing unit 102 , a first audio recording unit 104 , a second audio recording unit 106 , and the like.

[0058] The camera control unit 101 controls the shooting by the camera unit 10. The camera control unit 101 controls the camera unit 10 so that the camera unit 10 shoots the video with appropriate exposure based on the video obtained from the camera unit 10. In addition, the camera control unit 101 controls the camera unit 10 so that the main subject is focused based on the video obtained from the camera unit 10.

[0059] The image processing unit 102 outputs the video imaged by the imaging unit 10 to the display unit 16 in real time. As a result, the display unit 16 displays a live view.

[0060] The image processing unit 102 includes a correlating unit 102A, a first detecting unit 102B, an assigning unit 102C, and a recording unit 102D.

[0061] The association establishing unit 102A accepts the association establishment between the first microphone 12 and the person A and the association establishment between the second microphone 14 and the person B. Various methods can be used to accept the association establishment. For example, when the first microphone 12 is associated, the person A is displayed on the display unit 16 and the user touches and selects the person A, thereby the first microphone 12 is associated with the person A. Here, the association establishment means, for example, presetting to collect the audio of the person A via the first microphone 12.

[0062] The first detection unit 102B automatically detects the state of the subject while the video is being captured by the camera 1. The first detection unit 102B applies various technologies to recognize the state of the subject through image processing. For example, the first detection unit 102B processes the dynamic image data using the face recognition technology to recognize whether the person A and the person B are making a sound.

[0063] The assigning unit 102C assigns an identification code related to the volume adjustment of the audio data of each subject to the moving image data based on the result of the detection step. The assigned identification code is displayed when editing the moving image data, and the user can confirm the identification code.

[0064] The recording unit 102D records the moving image data output from the camera unit 10 in the storage unit 18. The moving image data may be recorded with the identification code assigned by the assigning unit 102C, or the moving image data before the identification code is assigned may be recorded in the storage unit 18. The recording unit 102D starts recording the moving image data according to an instruction from the user. And, it ends recording the moving image data according to an instruction from the user. The user instructs the start and end of recording via the operation unit 22.

[0065] The first audio recording unit 104 records the first audio data input from the first microphone 12 in synchronization with the moving image data in the storage unit 18. The first audio data is recorded in the storage unit 18 in association with the moving image data.

[0066] The second audio recording unit 106 records the second audio data input from the second microphone 14 in synchronization with the moving image data in the storage unit 18. The second audio data is recorded in the storage unit 18 in association with the moving image data.

[0067] Next, Figure 1 A specific example of obtaining dynamic image data of person A and person B described in is described below.

[0068] Figure 4 is a flow chart illustrating a data creation method implemented using the camera system 100 .

[0069] [Association Establishment Process]

[0070] In the association process, the user touches and specifies the person A displayed on the display unit 16 of the camera device 1, and the first microphone 12 is associated with the person A (step S10). Then, the user specifies the person B displayed on the display unit 16 of the camera device 1, and the second microphone 14 is associated with the person B (step S11).

[0071] [Video Process]

[0072] In the video recording process, the user starts recording the moving image data via the operation unit 22 (step S12). After that, the camera control unit 101 determines whether to continue recording the moving image data (step S20), and the video recording is performed until the user inputs an instruction to stop the video recording via the operation unit 22. On the other hand, if the user inputs an instruction to stop the video recording via the operation unit 22, the recording of the moving image data ends (step S21). In addition, during the video recording process, the recording process, the detection process, and the assignment process described below are performed.

[0073] [Recording Process]

[0074] In the recording process, the first audio data of the person A is recorded in the storage unit 18 by the first microphone 12, and the second audio data of the person B is recorded in the storage unit 18 by the second microphone 14 (step S13).

[0075] [Inspection process]

[0076] In the detection process, the first detection unit 102B detects that person A is making a sound (speaking) in the dynamic image data through image processing (step S14). In addition, in the detection process, the first detection unit 102B detects that person B is making a sound (speaking) in the dynamic image data through image processing (step S15). For example, the first detection unit 102B recognizes the faces of person A and person B using facial recognition technology, and detects whether person A and person B are speaking by analyzing the mouth images of person A and person B.

[0077] [Assignment process]

[0078] In the assigning process, when character A is not speaking, the assigning unit 102C assigns an identification code to relatively reduce the volume of the first audio data (recorded as the 1st MP in the figure) collected by the first microphone 12 to the dynamic image data (step S16). On the other hand, when character A is speaking, the dynamic image data is assigned an identification code to relatively increase the volume of the first audio data collected by the first microphone 12 (step S17). And, similarly, when character B is not speaking, the dynamic image data is assigned an identification code to relatively reduce the volume of the second audio data (recorded as the 2nd MP in the figure) collected by the second microphone 14 (step S18), and when character B is speaking, the dynamic image data is assigned an identification code to relatively increase the volume of the second audio data collected by the second microphone 14 (step S19). The dynamic image data to which the identification code is assigned is described below.

[0079] Figure 5 This is a diagram for explaining an example of moving image data to which an identification code is assigned.

[0080] The first detection unit 102B detects that the person A is speaking during the period from t1 to t2 in the moving image data. And, the assigning unit 102C assigns the identification code "1st microphone: loud" (reference numeral 130) for increasing the volume of the first microphone 12 to the moving image data based on the detection result of the first detection unit 102B. And, the first detection unit 102B detects that the person B is speaking during the period from t2 to t3 in the moving image data. And, the assigning unit 102C assigns the identification code "2nd microphone: loud" (reference numeral 132) for increasing the volume of the second microphone 14 based on the detection result of the first detection unit 102B. And, the first detection unit 102B detects that the person A is speaking during the period from t3 to t4 in the moving image data. And, the assigning unit 102C assigns the identification code "1st microphone: loud" (reference numeral 134) for increasing the volume of the first microphone 12 to the moving image data based on the detection result of the first detection unit 102B. Furthermore, in order to relatively increase the volume of the first microphone 12, the dynamic image may be given an identification code "Second microphone: low" instead of "First microphone: high". In addition, the identification code is not limited to the above-mentioned identification code, and various forms can be adopted as long as it indicates the volume adjustment of the first audio data and the second audio data. For example, as an identification code, "Second audio data: low" which reduces the volume of the second audio data may be added together with "First microphone: high". In addition, as an identification code, an identification code "First audio data: level 10" which marks the volume level of the first audio data may be added. In addition, the larger the numerical value of the volume level, the louder the volume.

[0081] [Video display process]

[0082] In the video display process, a video based on the recorded dynamic image data is displayed (step S22). The video based on the dynamic image data is displayed on a monitor of a computer provided separately from the camera device 1. For example, the user displays the video on the monitor to perform video editing. The user displays the video on the monitor and adjusts the volume of the first audio data and the second audio data. In addition, when the display unit 16 of the camera device 1 displays the video based on the dynamic image data to perform editing, the video can be displayed on the display unit 16 to perform editing.

[0083] [Volume adjustment acceptance process]

[0084] In the volume adjustment acceptance process, the volume adjustment based on the user's audio data is accepted (step S23). Specifically, the user adjusts the volume of the first audio data and / or the second audio data while confirming the dynamic image data displayed on the monitor and the identification code assigned to the dynamic image data. For example, when the user confirms Figure 5When the user uses the dynamic image data assigned with the identification code shown in FIG. 1 , the volume level of the first audio data is set to 10 and the volume level of the second audio data is set to 1 during the period from t1 to t2, thereby relatively increasing the volume of the first audio data. Furthermore, the user sets the volume level of the second audio data to 10 and the volume level of the first audio data to 1 during the period from t2 to t3, thereby relatively increasing the volume of the second audio data. Furthermore, the user sets the volume level of the first audio data to 10 and the volume level of the second audio data to 1 during the period from t3 to t4, thereby relatively increasing the volume of the first audio data.

[0085] As described above, in the data creation method of this embodiment, it is automatically detected whether the characters A and B are speaking in the dynamic image data through image processing, and an identification code related to volume adjustment is assigned to the dynamic image data according to the detection result. As a result, the user can confirm the identification code to adjust the volume of the first audio data and the second audio data when editing the dynamic image data, which can save the trouble of confirming the image again, and can effectively adjust the volume according to the status of the characters A and B.

[0086] In addition, in the above-mentioned embodiment, the hardware structure of the processing unit (image processing unit 102, camera control unit 101, first audio recording unit 104, second audio recording unit 106) that performs various processes is various processors (processors) as shown below. The various processors include a general-purpose processor that executes software (program) to function as various processing units, namely a CPU (Central Processing Unit), a processor that can change the circuit structure after manufacturing, such as a Programmable Logic Device (PLD), and a processor that has a circuit structure specially designed for performing specific processing, such as an ASIC (Application Specific Integrated Circuit), namely a dedicated circuit, etc.

[0087] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, multiple processing units may be composed of one processor. As an example of a method in which multiple processing units are composed of one processor, first, a method in which a processor is composed of a combination of one or more CPUs and software, such as a client and a server, and the processor functions as multiple processing units. Second, a method in which a processor that implements the functions of the entire system including multiple processing units through one IC (Integrated Circuit) chip is used, such as a system on chip (SoC) or the like. In this way, various processing units are composed of one or more of the above-mentioned various processors as a hardware structure.

[0088] Furthermore, the hardware structure of these various processors is, more specifically, a circuit that combines circuit elements such as semiconductor elements.

[0089] The above structures and functions can be appropriately implemented by any hardware, software or a combination of the two. For example, the present invention can also be applied to a program that causes a computer to execute the above processing steps, a computer-readable recording medium (non-transitory recording medium) recording such a program, or a computer capable of installing such a program.

[0090] <Second embodiment>

[0091] Next, a second embodiment of the present invention is described. In this embodiment, the volume of audio data synthesized with the dynamic image data is adjusted according to the state of the subject automatically detected from the dynamic image data. Thus, in this embodiment, dynamic image data with audio whose volume is adjusted according to the state of the subject can be effectively obtained.

[0092] Figure 6 This is a block diagram of the main functions implemented by the CPU when recording dynamic image data and audio data. Figure 3 The parts that have already been explained are marked with the same symbols and the explanation is omitted.

[0093] like Figure 6 As shown, the CPU 24 functions as an imaging control unit 101, an image processing unit 102, a first audio recording unit 104, a second audio recording unit 106, an adjustment unit 108, and a synthesis unit 110. In addition, the image processing unit 102 of this embodiment includes an association unit 102A, a first detection unit 102B, and a recording unit 102D.

[0094] The adjustment unit 108 automatically adjusts the volume of the first audio data recorded in the storage unit 18 and the second audio data recorded in the storage unit 18 based on the detection result of the first detection unit 102B. The adjustment unit 108 adjusts the volume of each audio data to a volume preset according to the state of the subject of the first detection unit 102B based on the detection result of the first detection unit 102B. In addition, the adjustment unit 108 can adjust the volume of the audio data before being synthesized by the synthesis unit 110, and can also adjust the volume of the audio data after being synthesized by the synthesis unit 110.

[0095] The synthesizing unit 110 synthesizes the dynamic image data and the audio data recorded in the storage unit 18 to generate dynamic image data with audio. The synthesizing unit 110 synthesizes the dynamic image data and the synchronized audio data to generate a video file. The file generated by the synthesizing unit 110 is in the form of a video file, for example, a file in the form of AVI, MP4, or MOV.

[0096] Figure 7 1 is a flowchart illustrating a data creation method implemented using the camera system 100. Figure 1 The specific example of obtaining the dynamic image data of the person A and the person B described in the description is described. Figure 4 The association establishment process, video recording process, audio recording process, and detection process that have been described are the same content, so the description is simplified.

[0097] [Association Establishment Process]

[0098] In the association process, the first microphone 12 and the person A and the second microphone 14 and the person B are associated (steps S30 and S31 ).

[0099] [Video Process]

[0100] In the video recording process, recording of the moving image data is performed, and the recording of the moving image data is terminated in accordance with the user's instruction (step S32, step S41, and step S42).

[0101] [Recording Process]

[0102] In the recording process, the first audio data and the second audio data are recorded in the storage unit 18 (step S33).

[0103] [Inspection process]

[0104] In the detection step, it is detected whether the person A is speaking in the moving image data (step S34). In addition, in the detection step, it is detected whether the person B is speaking in the moving image data (step S35).

[0105] [Adjustment process]

[0106] In the adjustment process, when the character A is not speaking, the volume of the first audio data is reduced by the adjustment unit 108 (step S36), and when the character A is speaking, the volume of the first audio data is increased by the adjustment unit 108 (step S37). Similarly, when the character B is not speaking, the volume of the second audio data is reduced (step S38), and when the character B is speaking, the volume of the second audio data is increased (step S39). The following is a specific description of the automatic adjustment of the volume of audio data.

[0107] Figure 8 This is a diagram for explaining volume adjustment of the first audio data and the second audio data.

[0108] During the period from t1 to t2 of the moving image data, the character A is speaking, so the adjustment unit 108 adjusts the volume of the first audio data to level 10. On the other hand, during the period from t1 to t2 of the moving image data, the character B is not speaking, so the adjustment unit 108 adjusts the volume of the second audio data to level 1. Furthermore, during the period from t2 to t3 of the moving image data, the character A is not speaking, so the adjustment unit 108 adjusts the volume of the first audio data to level 1. On the other hand, during the period from t2 to t3 of the moving image data, the character B is speaking, so the adjustment unit 108 adjusts the volume of the second audio data to level 10. Furthermore, during the period from t3 to t4 of the moving image data, the character A is speaking, so the adjustment unit 108 adjusts the volume of the first audio data to level 10. On the other hand, during the period from t3 to t4 of the moving image data, the character B is not speaking, so the adjustment unit 108 adjusts the volume of the second audio data to level 1. In addition, in the above description, the adjustment of the volume of the first audio data and the second audio data recorded in the storage unit 18 is described, but the present embodiment is not limited to this example. For example, the adjustment may be performed before the first audio data and the second audio data are recorded in the storage unit 18. In this case, the adjustment unit 108 is provided in the first audio recording unit 104 and the second audio recording unit 106.

[0109] [Synthesis process]

[0110] In the synthesis process, the synthesis unit 110 synthesizes the first audio data and the second audio data whose volume has been adjusted and the dynamic image data (step S40). For example, in the synthesis unit 110, the first audio data and the second audio data whose volume has been adjusted and the dynamic image data are synthesized to generate a video file in the AVI format.

[0111] As described above, the data creation method of this embodiment automatically detects whether the person A and the person B are talking in the dynamic image data, and adjusts the volume of the audio data according to the detection result. As a result, the user can effectively obtain the dynamic image data with audio in which the volume of the first audio data and the second audio data is adjusted according to the state of the subject of the dynamic image data without the user manually adjusting the volume.

[0112] Next, a modified example of the present invention is described. In the above description, an example of adjusting the volume according to whether the subject (person A and person B) is speaking is described. However, the application of the present invention is not limited to this example. Below, as a modified example, a case of adjusting the volume according to various states of the subject is described. In addition, the modified example described below is applicable to the above-mentioned method of assigning an identification code to dynamic image data (first embodiment) and the method of adjusting the volume of audio data (second embodiment).

[0113] <Variation 1>

[0114] Modification 1 is described. In this example, each subject has a position detection system, and the position of each subject is detected by the position detection system. In addition, an identification code for adjusting the volume of audio data is given or the volume of audio data is adjusted according to the detected position of each subject.

[0115] Fig. 9 1 is a block diagram showing a schematic structure of the camera system 100. Figure 2 The parts that have already been explained are marked with the same symbols and the explanation is omitted.

[0116] The first microphone 12 includes a first audio signal processing unit 12A, a first wireless communication unit 12B, and a first position detection system 12C. The first position detection system 12C detects the position of the first microphone 12. For example, the first position detection system 12C detects the position of the first microphone 12 through a GPS (Global Positioning System). Furthermore, since person A has the first microphone 12, the first position detection system 12C detects the position of person A. The position of person A detected by the first position detection system 12C is input to the imaging device 1 via the first wireless communication unit 12B and the third wireless communication unit 30. In addition, the second microphone 14 has the same structure as the first microphone 12, and therefore the description thereof is omitted.

[0117] Fig.10 2 is a block diagram of the main functions implemented by CPU 24 when recording dynamic image data and audio data. Figure 3 The parts that have already been explained are marked with the same symbols and the explanation is omitted.

[0118] like Fig.10 As shown, the CPU 24 functions as an imaging control unit 101 , an image processing unit 102 , a first audio recording unit 104 , a second audio recording unit 106 , and a second detection unit 112 .

[0119] The second detection unit 112 obtains information about the position of person A from the first microphone 12, and obtains information about the position of person B from the second microphone 14. Specifically, the second detection unit 112 obtains information about the position of person A detected by the first position detection system 12C of the first microphone 12, and obtains information about the position of person B detected by the second position detection system 14C of the second microphone 14. In addition, the second detection unit 112 detects the distance between person A and the imaging device 1, and the distance between person B and the imaging device 1.

[0120] The assigning unit 102C assigns an identification code related to the volume adjustment of the audio data of each subject to the dynamic image data based on the result of the second detection unit 112. For example, the assigning unit 102C assigns an identification code related to the volume adjustment of the first audio data based on the distance between the person A and the camera 1, and assigns an identification code related to the volume adjustment of the second audio data based on the distance between the person B and the camera 1. For example, when the distance between the person A and the camera 1 is greater than the distance α, the assigning unit 102C assigns an identification code for reducing the volume. Furthermore, when the distance between the person A and the camera 1 is within the distance β, the assigning unit 102C assigns an identification code for increasing the volume. Furthermore, as an adjustment of the audio data, for example, the first microphone 12 and the second microphone 14 may also reduce the volume in stages as the distance between the person A (or the person B) and the camera 1 becomes longer.

[0121] As described above, in this example, the position detection system acquires information related to the positions of person A and person B, and based on the information related to the positions, the second detection unit 112 accurately detects the positions of person A and person B. Furthermore, the volume can be effectively adjusted based on the positions of person A and person B.

[0122] <Variation 2>

[0123] Next, a modification example 2 will be described. In the modification example 2, an identification code is assigned or the volume of audio data is adjusted according to the direction the subject is facing.

[0124] In this example, the first detection unit 102B identifies the direction in which each subject is facing in the dynamic image data by image processing. For example, the first detection unit 102B uses facial recognition technology to identify the direction in which the person A and the person B are facing. In addition, an identification code is assigned or the volume of the audio data is adjusted according to the direction in which the person A and the person B are facing. For example, as an adjustment of the audio data, when the person A is facing the direction of the camera device 1 (front), the volume of the first audio data is increased, and when the person A is not facing the direction of the camera device 1, the volume of the first audio data is decreased.

[0125] Fig.11 This is a diagram for explaining a specific example of this example.

[0126] exist Fig.11 In the case shown in (A), person A faces forward relative to the camera 1. At this time, the first detection unit 102B detects that person A faces forward, and performs volume adjustment to increase the volume of the first audio data, which is audio data of person A. Fig.11 In the case shown in (B), person A faces sideways (not frontally) relative to the imaging device 1. At this time, the first detection unit 102B detects that person A faces sideways, and volume adjustment is performed to reduce the volume of the first audio data, which is audio data of person A.

[0127] As described above, in this example, the first detection unit 102B detects the direction in which the subject is facing, and effectively performs volume adjustment according to the direction in which the subject is facing.

[0128] <Variation 3>

[0129] Next, a modification example 3 will be described. In the modification example 3, an identification code related to volume adjustment of audio data is assigned or volume adjustment of audio data is performed according to the distance of the subject.

[0130] In this example, the first detection unit 102B recognizes the distance between each subject and the camera 1 in the dynamic image data by image processing. For example, the first detection unit 102B detects the distance between the person A and the person B and the camera 1 by the subject distance estimation technology based on image processing. In addition, an identification code is assigned or audio data is adjusted according to the distance between the person A and the person B and the camera 1. For example, as an adjustment of the audio data, when the distance between the person A and the camera 1 is greater than the threshold value γ, the volume of the first audio data is reduced.

[0131] Fig.12 This is a diagram for explaining a specific example of this example.

[0132] exist Fig.12In the case shown in (A), person A is located within the threshold value γ. At this time, the first detection unit 102B detects that person A is located within the threshold value γ, and the volume adjustment is performed to increase the volume of the first audio data, which is the audio data of person A. On the other hand, Fig.12 In the case shown in (B), person A is located farther than threshold value γ. At this time, the first detection unit 102B detects that person A is located farther than threshold value γ, and volume adjustment is performed to reduce the volume of the first audio data, which is audio data of person A.

[0133] As described above, in this example, the distance between the subject and the imaging device 1 is detected, and the volume adjustment is effectively performed according to the distance between the subject and the imaging device 1 .

[0134] <Variation 4>

[0135] Next, a modification example 4 will be described. In the modification example 4, an identification code is assigned or the volume of audio data is adjusted according to whether or not a subject exists in the field angle of the imaging device 1 .

[0136] In this example, the first detection unit 102B recognizes whether each subject exists within the field of view of the camera device 1 by image processing in the dynamic image data. For example, the first detection unit 102B uses image recognition technology to recognize whether person A and person B exist within the field of view of the camera device 1. And, an identification code is assigned or audio data is adjusted according to whether person A and person B exist within the field of view. For example, as an adjustment of the volume of audio data, when person A appears within the field of view of the camera device 1, the volume of the first audio data is increased, and when person A does not appear within the field of view of the camera device 1, the volume of the first audio data is decreased.

[0137] In addition, regarding the field of view angle of the camera device 1, for example, as described in Japanese Patent Gazette No. 2017-46355, when the field of view angle of the dynamic image data captured by the camera device 1 is different from the field of view angle of the dynamic image data actually stored in the storage unit 18, it refers to the field of view angle of the dynamic image data stored in the storage unit 18.

[0138] Fig.13 This is a diagram for explaining a specific example of this example.

[0139] exist Fig.13 In the case shown, person A is located within the field of view angle 151 of the camera 1, and person B is located outside the field of view angle 151. At this time, the first detection unit 102B detects that person A is located within the field of view angle 151, and the volume adjustment is performed to increase the volume of the first audio data, which is the audio data of person A. On the other hand, the first detection unit 102B detects that person B is not located within the field of view angle 151, and the volume adjustment is performed to decrease the volume of the second audio data, which is the audio data of person B.

[0140] As described above, in this example, the first detection unit 102B detects whether the subject is within the angle of view of the imaging device 1 , and effectively performs volume adjustment based on whether the subject is within the angle of view.

[0141] <Variant 5>

[0142] In this example, the camera 1 or the first microphone 12 and the second microphone 14 record audio data of stereo audio. The stereo audio includes audio for the left ear and audio for the right ear of a person. The first detection unit 102B recognizes whether the subject is on the left or right side relative to the center of the camera 1 in the dynamic image data through image processing, and assigns an identification code or adjusts the volume of the audio data. For example, as a method of adjusting the volume of the audio data, when a person is on the left side relative to the camera 1, the volume of the audio data for the left ear is relatively increased. In addition, as a method for identifying the position of a person, there are, for example, a method using image recognition technology and a method using GPS as in Modification 1.

[0143] Fig.14 This is a diagram for explaining a specific example of this example.

[0144] exist Fig.14 In the case shown in (A), person A is located on the L side with respect to the optical axis M of the camera 1. At this time, the first detection unit 102B detects that person A is located on the L side, and the left ear audio data in the first audio data, which is the audio data of person A, is relatively increased. Fig.14 In the case shown in (B), person A is located on the R side with respect to the optical axis M of the imaging device 1. At this time, the first detection unit 102B detects that person A is located on the R side, and the right ear audio data of the first audio data, which is audio data of person A, is relatively increased.

[0145] As described above, in this example, the first detection unit 102B detects on which side the subject is located relative to the imaging device 1 and adds a difference in volume between the left ear audio data and the right ear audio data, thereby producing more immersive moving image data with audio.

[0146] The first microphone 12 and the second microphone 14 may be a mobile phone or a smart phone. In this case, it is preferable that the mobile phone or the smart phone has an application for wirelessly connecting itself and the imaging device 1.

[0147] As mentioned above, although the example of this invention was demonstrated, this invention is not limited to the said embodiment, It is a matter of course that various deformation|transformation is possible within the range which does not deviate from the spirit of this invention.

[0148] Explanation of symbols

[0149] 1-camera device, 10-camera unit, 10A-camera optical system, 10B-camera element, 10C-image signal processing unit, 12-first microphone, 12A-first audio signal processing unit, 12B-first wireless communication unit, 12C-first position detection system, 14-second microphone, 14B-second wireless communication unit, 14C-second position detection system, 16-display unit, 18-storage unit, 20-audio output unit, 22-operating unit, 24-CPU, 26-ROM, 28-RAM, 30-third wireless communication unit, 100-camera system, 101-camera control unit, 102-image processing unit, 102A-association establishing unit, 102B-first detection unit, 102C-assignment unit, 102D-video recording unit, 104-first audio recording unit, 106-second audio recording unit, 112-second detection unit, A-person, B-person.

Claims

1. A data creation method, which is used in a camera system including an imaging device and a plurality of microphones connected to the imaging device, the data creation method comprising the following steps: an association establishing step of establishing an association between the microphones and the subjects having the microphones; A video recording step, using the camera device to record the dynamic image data; a recording step of recording the audio data of each of the photographed objects using each of the microphones in synchronization with the start of the video recording step; A detection process for automatically detecting the state of the subject during the video recording process; and and an assigning step of assigning, to the moving image data, an identification code related to volume adjustment of the audio data of each subject, based on a result of the detecting step.

2. The data creation method according to claim 1, wherein: In the detection step, the state in which the subject is making a sound is recognized by image processing in the dynamic image data. In the adding step, the identification code is added to the moving image data so as to increase the volume of the audio data of the subject making a sound relatively with respect to the volume of the audio data of other subjects.

3. The data creation method according to claim 1 or 2, wherein: In the detection step, the direction in which each subject is facing is identified in the dynamic image data by image processing. In the adding step, the identification code for adjusting the volume of the audio data according to the direction of the face of each subject relative to the imaging device is added to the moving image data.

4. The data creation method according to claim 1 or 2, wherein: In the detection step, the distance between each of the objects and the camera device is identified in the dynamic image data by image processing. In the adding step, the identification code for adjusting the volume of the audio data according to the distance of each subject is added to the moving image data.

5. The data creation method according to claim 1 or 2, wherein: In the detection step, whether each of the objects exists within the field of view of the camera device is identified by image processing in the dynamic image data. In the adding step, the identification code for adjusting the volume of the audio data according to whether the subject is within the angle of view of the imaging device is added to the moving image data.

6. The data creation method according to claim 1 or 2, comprising: a position acquisition step of acquiring the position of each of the objects from a position detection system of each of the objects; The detecting step detects the position of each subject obtained by the position acquiring step, In the adding step, the identification code related to the volume adjustment of the audio data of each subject is added to the moving image data based on the result of the detecting step.

7. The data creation method according to claim 1 or 2, wherein: After the providing step, the method includes a receiving step of receiving volume adjustment of the audio data by a user.

8. A recording medium which is non-temporary and computer-readable, wherein: When the instructions stored in the recording medium are read by a computer, the computer is caused to execute a data creation program, the data creation program being used in a camera system including an imaging device and a plurality of microphones located outside the imaging device and connected to the imaging device, the data creation program including the following steps: an association establishing step of establishing an association between the microphones and the subjects having the microphones; A video recording step, using the camera device to record the dynamic image data; a recording step of recording the audio data of each of the photographed objects using each of the microphones in synchronization with the start of the video recording step; a detection step, during the recording step, automatically detecting the state of the subject according to the dynamic image data; and and an assigning step of assigning, to the moving image data, an identification code related to volume adjustment of the audio data of each subject, based on a result of the detecting step.

Images