Photography application, camera-equipped mobile communication terminal, communication network system, and video generation system

Abstract

To provide a shooting application that enables video shooting in a specified format and allows easy acquisition of user information on a mobile device such as a smartphone in order to construct a virtual human.SOLUTION: A photographing application for acquiring voice characteristics of a user's face and speaking style has a video shooting function for shooting a video in a first designated format including at least the user's face direction, facial expression, shooting time, and whether a dialogue is read, and a dialogue reading video shooting function for shooting a dialogue reading video in a second designated format including at least the user's face direction, facial expression, number of sentences, and whether a pause is read.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a photography application, a communication network system, and a video generation system, and more particularly to a photography application that enables acquisition of user information suitable for a generation AI such as a virtual human. [Background technology]

[0002] Conventionally, when generating a virtual human for a user, the user would enter a large-scale 3D scanning device and perform a 3D scan to scan the entire body. Data on the user's head, ears, mouth, torso, legs, and other body parts would then be stored in a database, and a virtual human (avatar) would be generated from the stored image and voice data.

[0003] Since the generated virtual human reflects at least the user's facial image, voice, and movements, it is necessary to obtain information from the user and extract features. Although it is possible to generate a virtual human by obtaining information from the user, there is a demand for information acquisition means and methods that can be freely used in general-purpose usage scenarios, but such information acquisition means and methods have not yet been available.

[0004] Conventional photography applications aim to take beautiful photographs, so most of them detect the subject's face, smile, etc. to take beautiful pictures. There were no photography applications specialized in generating virtual humans. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2015-531098 [Patent Document 2] JP 2018-112814 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the above-mentioned Patent Document 1, although it is possible to obtain user information using a whole-body 3D scanner and a microphone, there is a problem in that the system cost is high.

[0007] In addition, while the above-mentioned Patent Document 2 does acquire personal information, no consideration was given to recording video of the user's information acquisition in a specified format suitable for machine learning for use in the virtual human generation AI.

[0008] The present invention has been devised in consideration of the above-mentioned problems, and its purpose is to provide a shooting application that enables video shooting in a specified format in order to construct a virtual human, and can easily obtain user information on a mobile terminal such as a smartphone.

[0009] Another object of the present invention is to provide a communication network system and a video generation system capable of generating a video of a user reading a new sentence using a machine learning model from a captured image and voice of the user. [Means for solving the problem]

[0010] The photography application of the first invention is a photography application for acquiring audio characteristics of a user's face and speaking style, characterized in that the photography application has a video shooting function that shoots a video in a first specified format including at least the user's facial direction, facial expression, shooting time, and whether or not to read lines, and a dialogue reading video shooting function that shoots a dialogue reading video in a second specified format including at least the user's facial direction, facial expression, number of sentences, and whether or not to pause.

[0011] The photography application according to a second invention is characterized in that, in the first invention, the first specification format is that the direction of the face is forward or up, down, left or right, and the facial expression is expressionless, and the video recording function records video by specifying conditions that combine the user's facial direction, facial expression, shooting time, and whether or not there is dialogue.

[0012] The photography application according to the third invention is characterized in that, in the first or second invention, the second specified format is that the direction of the face is forward, up, down, and left and right, and the facial expressions are neutral, anger, sadness, happiness, disgust, fear, and surprise, and the dialogue reading video shooting function shoots the dialogue reading video by specifying conditions that combine the user's facial direction, facial expressions, number of sentences, and whether or not there is a pause.

[0013] A camera-equipped mobile communication terminal according to a fourth aspect of the present invention is characterized in that the photography application according to the first aspect of the present invention is installed therein.

[0014] A communication network system according to a fifth aspect of the present invention is a communication network system in which a camera-equipped mobile communication terminal according to the fourth aspect of the present invention, a WEBRTC (Web Real-time Communication) server, and a machine learning server installed downstream of the WEBRTC server are connected via a communication network, wherein the WEBRTC server communicates in real time with a photography application installed on the camera-equipped mobile communication terminal, and registers audio data and video data in a first specified format including at least the user's facial direction, facial expression, shooting time, and whether or not dialogue is read, obtained by the video shooting function or the dialogue reading video shooting function, or a second specified format including at least the user's facial direction, facial expression, number of sentences, and whether or not pauses are performed, and performs voice recognition processing of the registered user's voice data and image analysis processing of the registered video data, learns the user's voice recognition processing results and image analysis results on the machine learning server, and stores a learning model.

[0015] The video generation system according to a sixth aspect of the present invention is characterized in that, in the communication network system according to the fifth aspect of the present invention, a virtual human or video of the user is generated by a machine learning model of the stored learning data. Effect of the Invention

[0016] According to the first to third inventions, it becomes possible to shoot video in a specified format in order to construct a virtual human, and a shooting application can be realized that can easily obtain user information on a mobile terminal such as a smartphone.

[0017] According to the fourth to sixth inventions, it is possible to realize a communication network system and a video generation system capable of generating a video of a user reading a new sentence using a machine learning model from a captured image and voice of the user. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram of a communication network system to which the present invention is applied. [Diagram 2] FIG. 2 is a functional block diagram of a shooting application according to the embodiment of the present invention. [Diagram 3] FIG. 3 is a diagram showing an example of a camera-equipped mobile communication terminal on which the image capturing application of FIG. 2 is installed. [Figure 4] FIG. 4 is a schematic process flowchart of the image capture application according to the embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram showing an example of the hardware configuration of camera-equipped portable communication terminal 1 of FIG. [Figure 6] FIG. 6 is a diagram showing an example of a scene photographed by a camera-equipped mobile communication terminal in which the photography application of FIG. 2 is installed. [Figure 7] FIG. 7 is a diagram showing an example of a user operation of the camera-equipped portable communication terminal of FIG. [Figure 8] FIG. 8 is a process flowchart of a display screen when a photographing application is started according to an embodiment of the present invention. [Figure 9] FIG. 9 is a diagram showing an example of the initial display screen of FIG. [Figure 10] FIG. 10 shows an example of the caution display screen of FIG. 8, where (a) shows caution 1 and (b) shows other caution examples. [Figure 11]FIG. 11 is a diagram showing an example of the shooting screen specification of FIG. [Figure 12] Fig. 12(a) is a diagram showing an example of a designation format when shooting a moving image, and Fig. 12(b) is a diagram showing an example of a designation format when shooting a moving image with dialogue reading. [Figure 13] FIG. 13 shows examples of screens showing changes in facial emotions, where (a) is an example image of anger, (b) is an example image of happiness, and (c) is an example image of surprise. [Figure 14] FIG. 14 is a flowchart of a video shooting process performed by the shooting application according to the embodiment of the present invention. [Figure 15] FIG. 15 is a functional block diagram of the WEBRTC server 2 in FIG. [Figure 16] FIG. 16 is a diagram illustrating an example of the hardware configuration of the WEBRTC server 2 in FIG. [Figure 17] FIG. 17 is a functional block diagram of the machine learning server 3 in FIG. [Figure 18] FIG. 18 is a diagram illustrating an example of the hardware configuration of the machine learning server 3 in FIG. [Figure 19] Fig. 19(a) is a diagram showing another example of a designation format when shooting a moving image, and Fig. 19(b) is a diagram showing another example of a designation format when shooting a dialogue reading moving image. [Figure 20] FIG. 20 is a diagram showing another example of a designation format when shooting a dialogue reading moving image. [Figure 21] FIG. 21 is a diagram showing another example of a designation format when shooting a dialogue reading moving image. [Figure 22] FIG. 22 is a flowchart showing detailed processing of moving image shooting by the shooting application according to the embodiment of the present invention. [Figure 23] Fig. 23(a) is a diagram showing an example of a shooting environment check page when shooting a dialogue-free video, and Fig. 23(b) is a diagram showing an example of setting values ​​of the shooting conditions in step 1401 in Fig. 22. [Figure 24] Fig. 24(a) is a diagram showing an example of a shooting environment check page when shooting a dialogue-free video, and Fig. 24(b) is a diagram showing an example of setting values ​​of the shooting conditions in step 1402 in Fig. 22. [Diagram 25] Fig. 25(a) is a diagram showing an example of a shooting environment check page for shooting a dialogue video, and Fig. 25(b) is a diagram showing an example of setting values ​​of the shooting conditions in step 1404 in Fig. 14 . [Figure 26] FIG. 26 is a diagram showing an example of dialogue when the dialogue moving image is shot in step 1405 of FIG. [Figure 27] FIG. 27 shows specific examples of the dialogue examples in FIG. 26, where (a) is an example of 5 patterns of 3-second sentences, (b) is an example of 5 patterns of 5-second sentences, and (c) is an example of 5 patterns of 10-second sentences. [Figure 28] FIG. 28 shows two examples of 15-second sentences and one example of one-minute sentence of the dialogue example in FIG. [Figure 29] Fig. 29(a) is a diagram showing an example of a shooting environment check page for shooting a dialogue video, and Fig. 29(b) is a diagram showing an example of setting values ​​of the shooting conditions in step 1406 in Fig. 22. [Diagram 30] Fig. 30(a) is a diagram showing an example of a shooting environment check page in which the face direction during dialogue video shooting is changed from the basic face image to up, down, left, right, etc. Fig. 30(b) is a diagram showing an example of the setting values ​​of the shooting conditions in step 1407 in Fig. 22. [Diagram 31] Fig. 31(a) is a diagram showing an example of a shooting environment check page in which the face direction during dialogue-free video shooting is changed from the basic face image to up, down, left, right, etc. Fig. 31(b) is a diagram showing an example of the setting values ​​of the shooting conditions in step 1408 in Fig. 22. [Diagram 32] Fig. 32(a) is a diagram showing an example of a shooting environment check page when the face direction is facing upward during dialogue video shooting, and Fig. 32(b) is a diagram showing an example of setting values ​​of the shooting conditions in step 1409 in Fig. 22. [Diagram 33] Fig. 33(a) is a diagram showing an example of a shooting environment check page when the face direction during dialogue video shooting is downwards. Fig. 33(b) is a diagram showing an example of setting values ​​of the shooting conditions in step 1410 of Fig. 22. [Diagram 34]Fig. 34(a) is a diagram showing an example of a shooting environment check page when the face direction during dialogue video shooting is turned leftward. Fig. 34(b) is a diagram showing an example of setting values ​​of the shooting conditions in step 1411 in Fig. 22. [Diagram 35] Fig. 35(a) is a diagram showing an example of a shooting environment check page when the face direction during dialogue video shooting is turned to the right. Fig. 35(b) is a diagram showing an example of setting values ​​of the shooting conditions in step 1412 in Fig. 22. [Diagram 36] FIG. 36 is a diagram showing an example of a shooting environment check page for checking the shooting environment while moving hands during dialogue moving image shooting. [Figure 37] Fig. 37(a) is a diagram showing an example of a shooting environment check page for the dialogue video shooting confirmation result, and Fig. 37(b) is a diagram showing an example of the confirmation result screen. [Figure 38] FIG. 38 is a diagram showing an example of an error display screen of the photography application. [Figure 39] FIG. 39 is a diagram showing another example of the error display screen of the photography application. [Diagram 40] FIG. 40 is a diagram showing an example of a login screen for the automatic moving image generation application in an embodiment of the present invention. [Diagram 41] FIG. 41 is a schematic flowchart of an automatic video generation application in an embodiment of the present invention. [Diagram 42] FIG. 42 is a diagram showing an example of a screen for creating a virtual human that resembles the user. [Diagram 43] FIG. 43 is a diagram (part 1) for explaining an example of creating a virtual human from the photograph in FIG. [Diagram 44] FIG. 44 is a diagram (part 2) for explaining an example of creating a virtual human from the photograph in FIG. [Diagram 45] FIG. 45 is a diagram for explaining an example of creating a virtual human from the photograph and voice of FIG. [Figure 46] FIG. 46 shows a list of the created virtual humans. [Figure 47]FIG. 47 is a diagram (part 1) showing an example of a moving image generation screen using the created virtual human. [Figure 48] FIG. 48 is a diagram (part 2) showing an example of a moving image generation screen using the created virtual human. [Figure 49] FIG. 49 is a diagram showing an example of a moving image generation completion screen. [Figure 50] FIG. 50 is a diagram showing an example of a moving image generation mode selection screen of an automatic moving image generation application in an embodiment of the present invention. [Figure 51] FIG. 51 is a schematic flowchart of an automatic moving image generation application that uses a virtual human generated by the photography application of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0020] <Embodiment>

[0021] The image capture application of this embodiment can be used by installing it in a user's camera-equipped mobile communication terminal or the like. As shown in Fig. 1, the image capture application connects a camera-equipped mobile communication terminal 1 and a WEBRTC server 2 to a network, extracts features of audio data and video data (moving image data) by capturing moving images in a format specified by the user through real-time communication, and generates a virtual human by performing learning using a machine learning model by a machine learning server 3 connected to the WEBRTC server 2. The learning results and the generated virtual human are stored in a database 4. This makes it possible to reduce the time and cost required to create realistic content using automatic moving image generation and various generation AIs using the stored virtual human.

[0022] Fig. 2 is a schematic functional block diagram of a photography application according to an embodiment of the present invention. This photography application is an application program for acquiring the voice characteristics of a user's face and speaking style, and is installed in camera-equipped portable communication terminal 1 for use. Fig. 3 shows an example of a display screen in which the photography application has been installed in camera-equipped portable communication terminal 1. The photography application can be started by touching photography application icon 20 with a finger or the like. Video shooting and audio input can be performed using camera 21 and microphone 22.

[0023] As shown in Fig. 2, the photography application 10 is composed of a video shooting function 11 using a first designated format and a dialogue reading video shooting function 12 using a second designated format. The video shooting function 11 shoots a video using a first designated format including at least the user's face direction, facial expression, shooting time, and whether or not dialogue is read. The dialogue reading video shooting function 12 shoots a dialogue reading video using a second designated format including at least the user's face direction, facial expression, number of sentences, and whether or not a pause is made. In the first designated format, the face direction is front or up, down, left, or right, and the facial expression is expressionless, and the video shooting function 11 shoots a video using conditions that combine the user's face direction, facial expression, shooting time, and whether or not dialogue is read. In the second designated format, the face direction is front, up, down, and left and right, and the facial expression is neutral, angry, sad, happy, disgust, fear, and surprise, and the dialogue reading video shooting function 12 shoots the dialogue reading video by specifying conditions that combine the user's face direction, facial expression, number of sentences, and presence or absence of pause. That is, the shooting application 10 shoots a video by the first designated format (step 801) or a dialogue reading video by the second designated format (step 802) as shown in Fig. 4. Further, as an extended function, a configuration may be made in which data items are added to ask about the user's past experiences, values, personality, hobbies, preferences, etc. Information on situations in which the user is in a natural state may also be acquired.

[0024] As shown in FIG. 1, the communication network system of the present embodiment is a communication network system in which a camera-equipped mobile communication terminal 1, a WEBRTC (Web Real-time Communication) server 2, and a machine learning server 3 installed behind the WEBRTC server 2 are connected via a communication network. The WEBRTC server 2 performs real-time communication with a shooting application installed in the camera-equipped mobile communication terminal 1, and registers voice data and video data of a first designated format including at least the user's face direction, facial expression, shooting time, and whether or not a line is read, acquired by a video shooting function 11 or a dialogue reading video shooting function 12, or a second designated format including at least the user's face direction, facial expression, number of sentences, and whether or not a pause is made, and performs voice recognition processing of the registered voice data of the user and image analysis processing of the registered video data, and learns the voice recognition processing results and image analysis results of the user on the machine learning server 3, and stores the learning data in a database 4. The communication network system of FIG. 1 also includes a video generation system that generates a virtual human or video of the user by a machine learning model of the learning data stored in the database.

[0025] FIG. 5 shows an example of the hardware configuration of the camera-equipped mobile communication terminal 1. As shown in FIG. 5, the camera-equipped mobile communication terminal 1 has a communication interface 40 for wireless communication such as LTE or WiFi (registered trademark), a processor 41 for controlling the entire system, a ROM 42, a RAM 43, a touch-type display 44, a soft keyboard 45 for inputting data from the screen of the display 44, a camera 46 for acquiring an image of the user's face, upper body, or whole body, a microphone 47 for acquiring voice information of the user, a speaker 48, a USB interface 49, and a rechargeable battery 50. These devices are connected to a system bus and exchange various data and control data. The camera-equipped mobile communication terminal 1 shown in FIG. 1 has a general hardware configuration, and smartphones such as iPhone (registered trademark) and Android (registered trademark) can be used. A tablet terminal or a notebook PC may be configured to have an LTE module built in and connect to a mobile communication network. As an example of the communication interface 40, WiFi (registered trademark) may also be used for communication.

[0026] The camera 46 is basically an in-camera for taking selfies, but since the camera-equipped mobile communication terminal 1 is a typical smartphone, it also has an out-camera. If the image sensor has about 2 million pixels or more, it can capture a high-definition image of the user. If the reality of the virtual human is required, a camera with higher resolution may be used. The microphone 47 may be a standard one, but an external microphone with a Bluetooth (registered trademark) interface may also be used.

[0027] For example, a lithium ion battery or a nickel metal hydride battery is used as the rechargeable battery 50. The USB interface 49 can be used for charging using an AC adapter via a connection cable, or for connecting an external device for data exchange, etc.

[0028] As shown in Fig. 6, user 30 shoots video by placing camera-equipped mobile communication terminal 1 with a shooting application installed on stand 9 on desk 31 and connecting cable 6 and AC adapter 7 plugged into outlet 8. As shown in Fig. 7, user 30 touches the display screen with his / her hand or the like to perform various video shooting operations.

[0029] Fig. 8 is a processing flowchart of the display screen when the photography application of the embodiment of the present invention is started. An example of the initial display screen of Fig. 8 is shown in Fig. 9. Examples of the notice display screen of Fig. 8 are shown in Figs. 10(a) and (b), with Fig. 10(a) showing notice 1 and Fig. 10(b) showing other notice examples. Fig. 11 is a diagram showing an example of the photography screen specifications of Fig. 8.

[0030] The specifications and operations of the photography application will be described below with reference to FIGS.

[0031] First, the photography application icon 20 is touched to start the photography application (step 801).

[0032] Next, when the user logs in to the WEBRTC (Web Real-time Communication) server 2 from the camera-equipped mobile communication terminal 1 (step 802), user information is acquired and authentication is performed. If the user is a legitimate user, the login is permitted. If user authentication cannot be performed, the login fails. User authentication is performed using a membership number ID, password, etc.

[0033] If login is successful, it is determined whether this is the first time the photo-taking app has been used (step 803). If this is the first time the photo-taking app has been used (step 803), an initial display screen such as that shown in Fig. 9 is displayed (step 804). In the example of Fig. 9, a title such as "DeepAICopy Photo-taking App" is displayed, and a message such as "This application will collect face and voice data necessary for creating your AI" is displayed.

[0034] Next, a caution display screen for the photography application as shown in Fig. 10(a)(b) is displayed (step 805). Caution 1 in Fig. 10(a) reads, "Do not vary the pitch or intonation of your voice." By sliding the screen to the right, good and bad examples are displayed. In Fig. 10(b), other cautions are displayed. For example, "Please prepare a stand to fix your smartphone. The entire process takes about 30 minutes. Even if you leave the photography application (smartphone) during the process, you can resume from there. If you register your email address, you will receive an email after learning is complete notifying you whether your copy AI (virtual human) was successfully created. The process may take some time, so do not press the button repeatedly. There are a total of five steps for photography (the number of steps is not limited to this), and you should press the buttons in the order of start recording → end recording → upload. Uploading may take some time." The contents of Fig. 10(b) are not limited to those shown, and may be changed as appropriate.

[0035] When the display of these screens has finished, the photographing screen is displayed as shown in Fig. 11 (step 806). On the photographing screen in Fig. 11, the face image area (frame) and background display area are displayed. The photograph is taken so that the face fits within the face image area (frame), but it does not matter if the face protrudes slightly. A simple background is preferable, but use a background that does not cause problems in face recognition and analysis. The large outer display frame in Fig. 11 is the maximum image acquisition area, and can display the upper body or the entire human body, with the background color being displayed in the remaining area.

[0036] The shooting screen specifications include a condition specification button for specifying shooting conditions, a camera preview screen, an area displaying the lines to be read, a button to move to the previous line, a button to move to the next line, an area displaying the current / total number of lines, a start recording button, an end recording button, an upload button, a done button, and an interrupt button. These buttons are displayed as necessary when shooting videos. In the example of Figure 11, the line to be read is displayed as "Based on statistics, the population of this town will double in five years." A message such as "Please read the above text aloud" is displayed. A start recording button, an end recording button, an upload button, and a button to go to the shooting page are displayed. When shooting a video with lines being read, the current number of lines and the total number are displayed.

[0037] In a shooting environment like that shown in Figures 6 and 7, the user reads prepared speech data, and the voice and video data showing the face are obtained. When the Complete or Interrupt button is pressed, information such as "STEPx completed" or "Interrupted at line y of STEPx" may be uploaded to the database. It may be possible to jump to the selected line of dialogue at once without having to tap the button each time. Voice recognition technology may be used to give instructions by voice commands, or finger gesture input may be used. It may be configured to check the volume of the captured voice and notify the user. The voice and video data are sent to the WEBRTC server using WEBRTC. The actual sound (voice) that was read is identified by the voice recognition section (voice recognition engine) of the machine learning server.

[0038] When the photographing is completed, the photographing result display screen is displayed (step 807).

[0039] FIG. 12(a) is a diagram showing an example of a designation format when shooting a video. FIG. 12(b) is a diagram showing an example of a designation format when shooting a video of a dialogue reading. FIG. 13(a), (b), and (c) show images expressing emotions of a user's face. For example, FIG. 13(a) is an example of an image expressing an emotion of "anger", FIG. 13(b) is an example of an image expressing an emotion of "happiness", and FIG. 13(c) is an example of an image expressing an emotion of "surprise". FIG. 14 is a flowchart of a video shooting process by a shooting application according to an embodiment of the present invention. Here, the setting value steps in FIG. 12(a) and (b) correspond to the processing steps in FIG. 14.

[0040] As shown in FIG. 12(a), the format for specifying the video shooting has a setting value for each shooting step, and data items of face direction, facial expression, shooting time, and presence or absence of dialogue. Other data items may be added to these data items. The reason for setting such a format for specifying the video shooting is that data suitable for machine learning can be obtained when generating (creating) a virtual human. Here, the setting values ​​for the shooting step 1001 are the face direction "directly in front", "expressionless", shooting time "30 seconds", and dialogue "none". Directly in front is not necessarily limited to directly in front, and it is sufficient as long as the image can be recognized as being directly in front. In addition, it may be configured to detect the face directly in front using image analysis technology. Expressionless means a state in which no emotion is expressed on the face. This is performed to obtain a facial image of the user in a normal state (normal state). The shooting time was set to 30 seconds, but a time that can be used for analyzing machine learning data was set. The setting values ​​for step 1002 are the face direction "up, down, left, right", "expressionless", shooting time "40 seconds", and dialogue "none". Since images of the face from above, below, left and right were to be obtained, 10 seconds was added to make the total 40 seconds.

[0041] As shown in FIG. 12(b), the designated format for shooting a dialogue reading video has a set value for each shooting step, and data items for face direction, facial expression, number of sentences, and presence or absence of pose. Other data items may be added to these data items. The reason for setting such a designated format for shooting a dialogue reading video is that data suitable for machine learning can be obtained, and by obtaining the number of sentences of the dialogue and pose data, a virtual human with a higher level of reality can be generated (created). This is different from normal camera shooting. Here, the set values ​​for the shooting step 1003 are the face direction "directly in front", "natural", the number of sentences "100 sentences", and the pose "none". The face direction is the same as in FIG. 12(a) for direct front, up, down, left, and right. The face direction is "free" literally means "free". This is because the user's individuality can be obtained by freely changing the face direction. The facial expression is "natural" meaning that the facial expression is natural, and is a concept close to expressionless. The number of sentences is set to 100, but the more sentences there are, the closer the virtual human to the user will be generated. When creating video content using a virtual human created by the generation AI, a data item for "pause" is set to distinguish between content that is better with pauses and content that is better without pauses.

[0042] The settings for the photographing step 1004 are face direction "up and down", facial expression "natural", number of sentences "10 sentences", pose "none", face direction "left and right", facial expression "natural", number of sentences "100 sentences", and pose "none". The "natural" facial expression is the same as step 1003. The reason for the difference in the number of sentences (10 sentences for up and down and 100 sentences for left and right) is that when the face direction is up and down, fewer features can be obtained even if the number of sentences is increased, so the number of sentences was reduced.

[0043] The settings for the photographing step 1005 are: face direction "front", expression "angry", number of sentences "50", pose "none", face direction "front", expression "sad", number of sentences "50", pose "none", face direction "front", expression "happy", number of sentences "50", pose "none". By adding emotions such as joy, anger, sadness and happiness to the facial expressions, it is possible to generate a virtual human that looks just like the user.

[0044] The settings for the photographing step 1006 are "free" for face direction, "free" for facial expression, "30 sentences" for the number of sentences, and "with pose." Here, "with pose" refers to, for example, reading lines while moving hands or fingers. "Free" facial expression is suitable for a virtual human generated by AI that expresses freely.

[0045] The video shooting process by the shooting application will be described below with reference to Figures 12(a) and 12(b) and 14. In the following, video shooting is performed with the setting values ​​set in Figures 12(a) and 12(b).

[0046] 12(a), a moving image is shot in which the user faces forward without any dialogue and with a blank expression (step 1001). Next, the angle of the user's face is specified and the moving image is shot (step 1002).

[0047] According to the settings in FIG. 12(b), a video of the person reading the lines is shot facing forward (step 1003).

[0048] Specify the face angle and shoot the dialogue reading video (step 1004).

[0049] A video of the lines being read with designated facial expressions is filmed (step 1005).

[0050] A video of the dialogue being read with upper body movement is recorded (step 1006).

[0051] The captured image is checked to see if it is correct, and then uploaded to the WEBRTC server (step 1007).

[0052] It may be configured to display an explanation screen (model) for shooting between the above steps.

[0053] Needless to say, the above shooting flow may be performed by changing the order of the shooting steps. However, it is better to perform the video shooting without a script and the video shooting with script reading together. This is because shooting them mixedly may affect the learning by the machine learning server.

[0054] <Configuration and Explanation of WEBRTC Server 2> FIG. 15 is a functional block diagram of the WEBRTC server 2 in FIG. 1.

[0055] As shown in FIG. 15, the WEBRTC server 2 includes a communication control unit 15, an audio data receiving unit 52, a video data receiving unit 53, a data transmitting unit 54, and a data transfer unit 55.

[0056] The communication control unit 51 performs real-time communication with the mobile communication terminal 1 with a camera via the WEBRTC API.

[0057] The audio data receiving unit 52 receives the user's audio data via the communication control unit 51.

[0058] The video data receiving unit 53 receives the user's video data via the communication control unit 51.

[0059] The data transmitting unit 54 transmits the voice recognition result processed by the subsequent machine learning server and the confirmation result of image data acquisition to the mobile communication terminal 1 with a camera.

[0060] The data transfer unit 55 transfers the received audio data and video data to the machine learning server 3.

[0061] <Hardware Configuration and Explanation of WEBRTC Server 2> FIG. 16 is a diagram showing a hardware configuration example of the WEBRTC server 2 in FIG. 1.

[0062] 16, the hardware of the WEBRTC server 2 includes a processor 66, a memory 67, a disk 68, a display 69, a keyboard 70, a mouse 71, a USB interface 72, a camera 73, a speaker 75, and a network interface 76. The WEBRTC server has the same configuration as a general server equipped with the WEBRTC API.

[0063] On the WEBRTC server side, an operator may be involved in the online connection, or a camera-equipped mobile communication terminal may be connected via an automatic response.

[0064] The processor 66 controls the entire system.

[0065] The memory 67 stores a WEBRTC server program.

[0066] Disk 68 stores the captured user's audio and video data.

[0067] The display 69 is an LED or organic EL display.

[0068] The keyboard 70 is used for inputting data.

[0069] The mouse 71 performs mouse operations on the server.

[0070] The USB interface 72 has an external connection interface and performs data backup and data exchange.

[0071] The camera 73 inputs an image of the operator's face and performs real-time communication with the user of the camera-equipped portable communication terminal 1 .

[0072] The microphone 74 receives voice input.

[0073] The speaker 76 outputs audio.

[0074] The network interface 76 is an interface for performing real-time communication with the network 5 .

[0075] <Configuration and explanation of machine learning server 3> FIG. 17 is a functional block diagram of the machine learning server 3 in FIG.

[0076] As shown in FIG. 17, the machine learning server 3 has a data communication unit 56, an image extraction unit 57, a shooting condition analysis unit 58, a facial image analysis unit 59, an expression emotion analysis unit 60, a voice recognition unit 61, a machine learning unit 62, a learning data storage unit 63, a learning data output unit 64, and a virtual human generation unit 65.

[0077] The data communication unit 56 connects to a WEBRTC server or other networks and performs data communication.

[0078] The image extraction unit 57 extracts images from the received video data.

[0079] The photographing condition analysis unit 58 performs a photographing condition analysis on the images extracted by the image extraction unit 57 .

[0080] The face image extracted by the image extraction unit 57 is analyzed.

[0081] The facial expression emotion analysis unit 60 acquires user emotion information such as joy, anger, sorrow, and happiness from the acquired facial expression emotion image.

[0082] The voice recognition unit 61 performs voice recognition of the lines read by the user. The machine learning unit 62 uses a machine learning model to learn the user's voice and speaking style.

[0083] The learning data storage unit 63 stores the learning results.

[0084] The learning data output unit 64 stores the learning results in a database.

[0085] The virtual human generation unit 65 generates a virtual human for the user from the results of learning by the machine learning unit 62.

[0086] <Hardware configuration and explanation of machine learning server 3> FIG. 18 is a diagram illustrating an example of the hardware configuration of the machine learning server 3 in FIG.

[0087] As shown in FIG. 18, the machine learning server 3 includes a processor 77, a memory 78, a storage 79, a server collaboration unit 80, an input / output interface 81, and a communication interface 82.

[0088] The processor 77 controls the entire system.

[0089] The memory 78 stores the machine learning server program and various data.

[0090] Storage 79 stores machine learning server programs, machine learning models, and various audio data and video data.

[0091] The server interworking unit 80 cooperates with the WEBRTC server.

[0092] The input / output interface 81 has various input / output interfaces, and connects a display, a keyboard, a mouse, and the like.

[0093] The communication interface 82 includes a network interface and a LAN interface.

[0094] Fig. 19(a)(b), Fig. 20, and Fig. 21 are diagrams showing detailed examples of the specification format when shooting a video. The specification format is similar to that in Fig. 12(a)(b), so the explanation will be omitted and only the added content will be explained. The difference is that "disgust," "fear," and "surprise" have been added to the emotional expressions of facial expressions. Various types of video shooting are performed according to these, so the number of data items for setting values ​​has increased. This makes it possible to generate a virtual human that is closer to the user himself.

[0095] FIG. 19(a) is a diagram showing an example of a designation format when shooting a video. FIG. 19(b), FIG. 20, and FIG. 21 are diagrams showing an example of a designation format when shooting a dialogue reading video. FIG. 22 is a flowchart showing a detailed video shooting process by a shooting application according to an embodiment of the present invention. Here, the setting value steps in FIG. 19(a)(b), FIG. 20, and FIG. 21 correspond to the processing steps in FIG. 22. Hereinafter, the detailed video shooting process flow by the shooting application will be described with reference to FIG. 22.

[0096] First, a video is taken of the user facing forward with a neutral expression (step 1401). The shooting environment check page and settings for this processing step 1401 are shown in Fig. 23(a)(b). Here, the user looks directly into the camera. They can breathe normally with their mouth closed and blink naturally. The WEBRTC server determines whether the face is captured in the video and whether the facial expression is neutral, and the results can be checked on the shooting result display screen. OK / NG is also displayed at the bottom of the shooting screen.

[0097] Next, a video with a different facial expression is shot from the front with a neutral expression (step 1402). The shooting environment check page and setting values ​​for this processing step 1402 are shown in Figure 24 (a) (b). Here, the user gazes directly at the camera. With their mouth closed, they can breathe normally and blink naturally. The facial expression changes from a neutral state to a smiling expression over five seconds, remains the same for ten seconds, and then changes back from the smiling state to a neutral expression over five seconds. The above movements are performed not only for smiling expressions, but also for expressions of joy, anger, sadness, and happiness. The WEBRTC server determines whether the face is captured in the shot video and whether the facial expression is neutral, and the results can be checked on the shooting result display screen. OK / NG is also displayed at the bottom of the shooting screen.

[0098] Next, a moving image is shot with a neutral facial expression and a specified face angle (step 1403).

[0099] Next, the line reading video is filmed while facing forward in a natural pose (step 1404). The filming environment check page and setting values ​​for this processing step 1404 are shown in Figures 25(a) and (b). It is preferable to be natural while filming, and you can freely change the direction of your face without being conscious of the camera. This makes it possible to obtain a video with natural movements. The voice recognition results of the read sentence can be confirmed at the bottom of the filming screen. OK / NG is displayed on the filming result display screen.

[0100] Next, a natural conversation scene is photographed (step 1405). As shown in FIG. 26, the following sentence is read and photographed as if the person is speaking naturally. 3-second sentences x 5 patterns The text is provided here (the text is on the next page) 5-second sentences x 5 patterns We provide the text 10-second sentences x 5 patterns We provide the text 15-second sentences x 2 patterns About your favorite ○○ 1 minute sentence x 1 pattern Self-introduction, etc., content that you can speak fluently for 1 minute without looking at the script

[0101] As a specific example, as shown in FIG. 3-second sentences x 5 patterns (1)Hello. It's hot today. (2)What is your favorite food? (3) I went to Kamakura yesterday. (4) Practice starts again tomorrow. (5) I bought new glasses.

[0102] Specifically, as shown in FIG. · 5 second sentences x 5 patterns (1)Is eating three meals a day good for your body? (2) A beautiful morning with the wind dancing through the flower fields and the birds singing. (3) As the rain taps on the window, I am immersed in the world of the book. The wings of imagination. (4) Old bridges tell stories, and cobblestones bear the mark of history. (5) The sunset seen from the top of the mountain turns the earth orange, and a sense of peace spreads.

[0103] Specifically, as shown in FIG. · 10 second sentences x 5 patterns (1) The spring breeze caresses the skin. Cherry blossoms dance fleetingly, and the anticipation of a new season smiles upon the heart. It is a sign of hope. (2) The roads we travel are filled with surprises and discoveries. Unfamiliar scenery adds new colors to our hearts and expands the appeal of adventure. (3) The sound of music stirs emotions, resonating in the heart with passion and emotion that cannot be expressed in words. (4) As the sunset dyed the sky orange and purple, signaling the end of the day, a feeling of gratitude and peace seemed to fill my heart. (5) Life is a series of challenges and growth, and failure is a stepping stone to success. The courage and effort required to pursue your dreams are the source of a bright future.

[0104] Specifically, as shown in FIG. 15-second sentences x 2 variations About your favorite ○○ (1) Friendship is a treasure that grows over time and deepens bonds. Friends who laugh, cry, and support each other enrich life and give it joy and meaning. Even when we are far apart, our hearts are connected. (2) A new day holds infinite possibilities and brings hope and courage, just as the light of the morning sun illuminates the darkness of the night. We grow by taking a step into the future while keeping in mind the lessons of the past. 1 minute sentence x 1 pattern Self-introduction, etc., 1 minute of content that you can speak fluently without looking at the script (omission)

[0105] Next, a video of the dialogue being read is shot with a specified frontal facial expression (step 1406). The shooting environment check page and settings for this processing step 1406 are shown in Figures 29(a) and (b). The number of sentences to be read here is 300 sentences in total, 50 sentences for each facial expression (anger, sadness, happiness, disgust, fear, surprise). Images are extracted from the shot video, and the WEBRTC server side determines whether the facial expressions express the specified emotions. The voice recognition results for the read sentences and whether the shooting conditions are OK or not can be confirmed on the shooting result display screen.

[0106] Next, a video of the line reading is shot with a specified frontal facial expression (step 1407). The shooting environment check page and setting values ​​for this processing step 1407 are shown in Figures 30(a) and (b). In Figure 30(a), after clicking the "Upload" button in the blank screen on the right, if there are no problems with the shooting results, "OK" is displayed, and if there are problems, an error message such as "Speech is interrupted" is displayed.

[0107] Here, the orientation of the face to be photographed is one pattern: front → up / down → left / right → front and back. The angle of the face photographed is judged to be appropriate. The facial expression during the shoot is natural. 10 sentences are read when the face is facing up / down, and 100 sentences when the face is facing left / right. The voice recognition results of the read sentences can be confirmed at the bottom of the shooting screen. The OK / NG status of the voice recognition and shooting conditions can also be confirmed on the shooting result display screen.

[0108] Next, a video of the line reading is shot with a blank expression and the angle of the face specified (step 1408). The shooting environment check page and setting values ​​for this processing step 1408 are shown in Figure 31 (a) (b). Here, the angle at which the face should be tilted up, down, left and right is approximately 45 degrees. The facial expression during shooting is expressionless (mouth closed). One pattern for facial movement is a path such as front → up and down → left and right → front, and all patterns are shot. The shooting time is 10 seconds x 4 patterns, for a total of 40 seconds. The WEBRTC server side judges whether the face is visible and whether the facial angle is appropriate, and the voice recognition and shooting conditions can be confirmed as OK / NG on the shooting result display screen.

[0109] Next, an upward facial expression is specified and a video of the lines being read is shot (step 1409). The shooting environment check page and setting values ​​for this processing step 1409 are shown in Fig. 32(a)(b). The number of sentences to be read is 50 for each facial expression (anger, sadness, happiness, disgust, fear, surprise), for a total of 300 sentences. Images are extracted from the shot video, and the WEBRTC server side judges whether the facial expression matches the one expressing the specified emotion. The voice recognition results of the read sentences and whether the shooting conditions were OK or not can be confirmed on the shooting result display screen.

[0110] Next, a downward facial expression is specified and a video of the lines being read is shot (step 1410). The shooting environment check page and setting values ​​for this processing step 1410 are shown in Fig. 33(a)(b). The number of sentences to be read is 50 for each facial expression (anger, sadness, happiness, disgust, fear, surprise), for a total of 300 sentences. Images are extracted from the shot video, and the WEBRTC server side determines whether the facial expressions express the specified emotions. The voice recognition results for the read sentences and whether the shooting conditions are OK or not can be confirmed on the shooting result display screen.

[0111] Next, a left-facing facial expression is specified and a video of the lines being read is shot (step 1411). The shooting environment check page and setting values ​​for this processing step 1411 are shown in Fig. 34(a)(b). The number of sentences to be read is 300 sentences in total, 50 sentences for each facial expression (anger, sadness, happiness, disgust, fear, surprise). Images are extracted from the shot video, and the WEBRTC server side judges whether the facial expression matches the specified emotion. The voice recognition results of the read sentences and whether the shooting conditions are OK or not can be confirmed on the shooting result display screen.

[0112] Next, a right-facing facial expression is specified and a video of the lines being read is shot (step 1412). The shooting environment check page and setting values ​​for this processing step 1412 are shown in Fig. 35(a)(b). The number of sentences to be read is 300 sentences in total, 50 sentences for each facial expression (anger, sadness, happiness, disgust, fear, surprise). Images are extracted from the shot video, and the WEBRTC server side judges whether the facial expression matches the specified emotion. The voice recognition results of the read sentences and whether the shooting conditions are OK or not can be confirmed on the shooting result display screen. In step 1413, the photographing results are checked to see if they are correct, and are then uploaded. This processing step 1413 is performed each time photographing is performed in each of steps 1401 to 1412.

[0113] Fig. 36 shows a user reading lines while moving their hands or fingers. In the example of Fig. 36, it says, "Our products are installed in auxiliary circuits that are essential for vehicle operation, integrating circuits for air conditioning and heating control, lighting control, inverter control, etc., as well as heaters and other railroad vehicle control devices." This makes it possible to generate a virtual human with improved presentation expressiveness.

[0114] A configuration may be adopted in which an explanation screen (example) for photographing is displayed between each of the above steps.

[0115] It goes without saying that you can change the order of the steps in the above shooting flow. However, it is better to shoot videos without dialogue and videos with dialogue together. Mixing the two types of shooting can affect the learning process of the machine learning server.

[0116] In this way, the captured voice data and video data are subjected to voice recognition, facial presence / absence recognition, facial expression estimation, and face angle estimation, and the results are displayed in a table like those in Figure 37(a)(b). Images that are judged as NG can be re-shot. Displaying all 500 images would be too many, so it is also possible to display only those that were NG. Once re-shooting is complete, the video data is uploaded to the server.

[0117] The initial screen will not be displayed from the next time, so if it is the first shooting, a message will be sent to the database that "the first shooting is over."

[0118] 38 and 39 show examples of error display screens. In the example of FIG. 38, an error message saying "Could not hear speech" is displayed. In FIG. 38, "3 / 100" is displayed, so it is clear that there are 100 documents of lines to be read, and an error occurred when reading the third line. Therefore, normal audio data and recorded data can be obtained by pressing the "back" button and re-recording. Also, in the example of FIG. 39, an error message saying "Please move away from the screen a little" is displayed. In FIG. 39, "1 / 100" is displayed, so it is clear that there are 100 documents of lines to be read, and an error occurred when reading the first line. As in the above, it is possible to avoid errors and record video in an appropriate position.

[0119] The current and total numbers in Figures 38 and 39 are displayed on the screen in the same way when shooting other videos. The blank spaces on the display screens in Figures 31(a), 32(a), 33(a), 34(a), 35(a), and 36(a) can be used to display error messages as explained in Figure 30(a). Also, the display screen does not necessarily have to have a two-screen configuration. If the screen size is small, a single screen will suffice.

[0120] An example of automatic video generation by an automatic video generation application using a virtual human generated (created) by a photography application (Deep AI photography application) in this embodiment will be described.

[0121] A user is registered in advance and uses the automatic video generation app. If the user is not registered, the user must register the company and the user before using the service. The service provider can provide this service as a cloud service or on a standalone server.

[0122] For example, in the case of a cloud service, when a URL sent by the service provider via email is clicked, a login screen is displayed as shown in Fig. 40, and login processing is performed. For example, a registered user name and password are entered, and once user authentication is approved, the user can connect to the automatic video generation application. In the example of Fig. 40, in addition to the login menu, a company registration menu and a user registration menu are displayed in order for the service provider to manage users.

[0123] 41 is a schematic flowchart of an automatic video generation application in an embodiment of the present invention. The process flow of the automatic video generation application will be described below.

[0124] First, the automatic moving image generation application executes a virtual human generation step for creating a virtual human that looks exactly like the user and moves and talks (step 4001). Here, the virtual human reflects at least the facial image, voice, and movements of the user.

[0125] Next, an input step is executed in which a sentence to be read by the virtual human generated in the virtual human generation step is input (step 4002).

[0126] Next, a setting step is executed to set a background image and / or a video (step 4003).

[0127] Next, a selection step is executed to select a virtual human to be used for creating a video from the multiple virtual humans created in the virtual human generation step (step 4004). Here, if a virtual human generated (created) by the photography application (Deep AI photography application) of this embodiment is stored in a database in advance, that virtual human can be used, and the virtual human generation step by the automatic video generation application can be simplified. Virtual humans can be used in a variety of applications, such as responding to inquiries, advertising, real estate, and call centers. Appropriate virtual humans can be generated according to these usage scenarios, expanding the range of applications.

[0128] A moving image generating step is executed (step 4005) for automatically generating a moving image from the background image and / or moving image set in the setting step 4003 and the virtual human selected in the selection step 4004. Here, before generating the moving image in the moving image generating step, one or more pieces of information may be input from among moving image title information, subtitle information to be superimposed on the moving image, and language selection information for converting voice into a foreign language.

[0129] 42 is a diagram showing an example of a virtual human creation screen that creates a virtual human that looks just like the user. Hereinafter, virtual human creation will be described in detail with reference to FIGS.

[0130] The example in Figure 42 shows an example of a screen when the virtual human creation menu is selected on the main menu screen of an automatic video generation application. This is an example of an explanatory screen for creating a virtual human on the virtual human creation page. The main menu includes video creation, download page, virtual human creation, virtual human purchase, billing information, fees, usage, password change, logout, etc.

[0131] <Virtual human creation page> Here you can create a virtual human from photos, audio or video.

[0132] How to use <Create a virtual human from a photo> 1. Enter a name for the virtual human you are creating.

[0133] Virtual human names can only contain lowercase English letters and numbers. The first character of a virtual human name cannot be a number.

[0134] 2. Choose the voice quality from male or female.

[0135] 3. Select an image. Follow the instructions in the selection box to select or take a sample image.

[0136] 4. Press the Create Virtual Human button.

[0137] <Create a virtual human from photos and audio> 1. Enter a name for the virtual human you are creating.

[0138] Virtual human names can only contain lowercase English letters and numbers. The first character of a virtual human name cannot be a number.

[0139] 2. Choose the voice quality from male or female.

[0140] 3. Select an image. Follow the instructions in the selection box to select or take a sample image.

[0141] 4. Please register at least three sentences for the reading audio. 1) Press the Rec button to start recording, read into the microphone, and then press the Stop button to stop recording. 2) Press the Check button to check if the recording was successful. *If the recording is successful, "No problem" will be displayed. 3) If there is a problem with the recording, read again so that "No problem" is displayed for at least three sentences.

[0142] 5. Press the Create Virtual Human button.

[0143] FIG. 43 is a diagram (part 1) for explaining an example of creating a virtual human from the photograph in FIG. 42. The example in FIG. 43 shows an example of taking a photograph yourself after reading the explanation of how to use the virtual human creation page in FIG. 42. Here, color and other factors do not matter as long as there are no people or other objects in the background when taking the photograph. A virtual human can be created by taking only one photograph. A virtual human may also be created by taking multiple photographs. The photograph is taken by placing the subject in the field of view of a smartphone camera or digital camera and pressing the start shooting button to complete the photographing process.

[0144] Fig. 44 is a diagram (part 2) for explaining an example of creating a virtual human from the photograph in Fig. 42. It is possible to take your own photograph or use a prepared photograph. Image files can be appropriately used, for example, general image files such as JPEG files, GIF files, PNG files, and the like, or EXIF ​​files from digital cameras.

[0145] FIG. 45 is a diagram for explaining an example of creating a virtual human from the photo and voice of FIG. 42. The example of FIG. 45 shows an example of creating a virtual human from a photo and voice. In this case, after steps 1, 2, and 3 for creating a virtual human from a photo, step 4 is additionally performed to record the voice. Here, an example of registering three sentences is shown. First sentence: "According to statistics, the population of this town will double in five years." is spoken into the microphone and registered with the Rec button. Voice registration is stopped with the Stop button. Press the Check button to check, and if it is normal, voice registration of the second sentence is performed. Second sentence: "It is said that this is the first time in the world that a forecast can be made up to 30 minutes in real time." is spoken, and the same operation as the first sentence is performed. Third sentence: "La Vie en Rose is a type of Edith Piaf." is registered with voice. The registration operation is as described above. In the voice registration of FIG. 45, the user's own voice can be recorded and his / her own voice data can be created. The voice is reflected in the artificial intelligence (Deep AI). Once you have registered your voice and clicked the Create Virtual Human button, you can create a virtual human linked to your photo and voice.

[0146] You can also add more voice recordings by pressing the Add Sentence button. The service specifications allow for 10 sentences, but you can add more than this. The benefit of this is that the accuracy is higher when there is a lot of voice data.

[0147] A virtual human can also be created by extracting still images from a video. A virtual human can also be created by taking pictures in a studio. When taking pictures in a studio, a more accurate virtual human can be created.

[0148] Figure 46 shows a list of created virtual humans. In the above-described virtual human creation, any virtual human created up to now can be called up at any time. Here, V1, V2, and V3 indicate that billing information differs depending on the type of virtual human in the cloud service. Virtual humans are classified by gender, such as male or female, and those that have already been generated and registered can be used by the owner of that virtual human.

[0149] Fig. 47 is a diagram (part 1) showing an example of a moving image generation screen using the created virtual human. Figs. 48 to 50 are diagrams for explaining the moving image generation process. Fig. 51 is a detailed flowchart of an automatic moving image generation application in an embodiment of the present invention. Moving image generation will be explained below with reference to Figs. 47 to 51.

[0150] First, the automatic video generation application executes a step of selecting a video generation mode (step 4101) as shown in FIG. 47 or FIG. 50. There are three types of video generation modes: leave it to AI, generate by specifying background and BGM, and create an explanatory text with ChatGPT. For example, as shown in FIG. 47, let us assume that you select leave it to AI.

[0151] Next, a step of inputting a video title is executed (step 4102). For example, "Weather in Tokyo in 2023".

[0152] Next, input the sentence you want the AI ​​to read (step 4403). For example, the weather in Tokyo in 2023 will be sunny.

[0153] If you check Add subtitles to video, subtitles will be automatically generated for the video (step 4404). The cloud service specifications state that the number of characters that can be entered is 23, but this is not limited to this. Subtitles are optional, so if you do not want to add subtitles to the video, uncheck the box. If you select Leave it to AI, you will not be able to set background music. Setting background music will increase the data volume, and in order to shorten the video generation time, background music setting is omitted. If a sentence is long, there is a risk that natural language analysis will not be possible, so if more than 180 characters are written without punctuation, the video may not be generated properly, so input 180 characters or less.

[0154] Next, the user selects a virtual human that will read the text (step 4105). Figure 47 shows an example in which the default HAL Version:x is selected. The user can also create a virtual human that looks just like him or herself.

[0155] Next, the language of the audio output of the video to be generated is selected (step 4106). The specifications allow automatic translation into Japanese, English, and Chinese, but are not limited to these languages. Translation into other languages ​​is also possible by installing a translation engine for other languages. The example in FIG. 30 shows a setting for Japanese to Japanese. When translating into other languages ​​and generating a video, it costs 1 yen per 30 characters. The price is set to be rounded up.

[0156] When the above steps are completed and a button to open a preview screen is clicked (step 4108), automatic generation of the video begins (see FIG. 47), and when video generation is completed (step 4108), a video preview screen like that shown in FIG. 48 is displayed. The generation time and generation rate are displayed. In the example of FIG. 48, it takes 47 minutes to generate the video, and the generation rate is 100% complete. These steps are performed by automatic video generation on a browser basis, but care should be taken not to generate videos simultaneously from multiple browsers, as this may cause the created video to become distorted. After the video is generated, a video download link is displayed, and the created video can be downloaded by clicking the download link. The created video is saved as an image file such as MP4.

[0157] In addition, when the generated video is played back, if a voice mistake (misspeak) or translation error occurs, a correction and update function may be provided so that the video can be corrected and updated. [Explanation of symbols]

[0158] 1. Camera-equipped mobile communication device 2 WEBRTC Server 3. Machine Learning Server 4 Database 5. Network 6 Cable 7 AC adapter 8 Power outlet 9 Stand Base 10. Photography Applications 11. Video recording function using the first specified format 12. Dialogue reading video recording function using the second specified format 20 Photo app icon 21 Camera (in-camera) 22. Mike 30 users 31 desk 32 User's hand (fingers) 40 Communication Interface 41 Processors 42 ROM 43 RAM 44 Display 45 Soft Keyboard 46 Camera 47. Mike 48 Speaker 49 USB Interface 50 Battery (rechargeable battery) 51 Communication control section 52 Audio data receiving unit 53 Video data receiver 54 Data transmission unit 55 Data transfer section 56 Data Communications Department 57 Image Extraction Section 58 Shooting Condition Analysis Section 59 Facial Image Analysis Department 60 Facial Emotion Analysis Department 61 Voice Recognition Unit 62 Machine Learning Department 63 Learning data storage unit 64 Learning data output unit 65 Virtual Human Generation Unit 66, 77 processors 67, 78 Memory 68 Discs 69 Display 70 Keyboard 71 Mouse 72 USB interface 73 Camera 74 Mike 75 Speaker 76 Network Interface 79 Storage 80 Server Linkage Department 81 Input / Output Interface 82 Communication Interface 4001: Avatar creation step 4002: Input step 4003: Configuration step 4004: Selection step 4005: Video generation step 4101: Video generation mode selection step 4102: Video title input step 4103: Steps for entering text you want to read 4104: Subtitle input step 4105: Avatar input step for reading text aloud 4106 : Language selection step 4107: Preview screen display step 4108: Video generation completion step 4109: Video download link display step

Claims

1. 1. A photography application for capturing facial and speaking audio characteristics of a user, comprising: The photography application a video capture function for capturing a video in a first designated format including at least the user's facial direction, facial expression, capture time, and whether or not the user is reading lines; and a dialogue reading video shooting function for shooting a dialogue reading video in a second specified format including at least the user's facial direction, facial expression, number of sentences, and presence or absence of pauses.

2. the first designated format is a facial orientation that is frontal, up, down, left, or right, and a facial expression that is expressionless; 2. The image capture application according to claim 1, wherein the video capture function captures video under conditions specified by a combination of the user's facial direction, facial expression, capture time, and the presence or absence of dialogue.

3. The second designation format is a format in which the face direction is front, up, down, and left and right, and the facial expression is a format in which the face direction is neutral, angry, sad, happy, disgust, fear, and surprise, 3. The image capture application according to claim 1, wherein the dialogue reading video capture function captures a dialogue reading video under conditions specified by a combination of the user's face direction, facial expression, number of sentences, and presence or absence of a pause.

4. 2. A camera-equipped mobile communication terminal having the image capturing application according to claim 1 installed thereon.

5. A communication network system in which the camera-equipped mobile communication terminal according to claim 4, a WEBRTC (Web Real-time Communication) server, and a machine learning server installed behind the WEBRTC server are connected via a communication network, The WEBRTC server performs real-time communication with a photography application installed on the camera-equipped mobile communication terminal, and registers voice data and video data in a first designated format including at least the user's facial direction, facial expression, shooting time, and whether or not dialogue is read, which are acquired by the video shooting function or the dialogue reading video shooting function, or in a second designated format including at least the user's facial direction, facial expression, number of sentences, and whether or not a pause is performed, and performs voice recognition processing of the registered user's voice data and image analysis processing of the registered video data, A communication network system characterized in that the machine learning server learns the user's voice recognition processing results and image analysis results, and stores a learning model.

6. 6. The communication network system according to claim 5, wherein the moving image generating system generates a virtual human or a moving image of the user by a machine learning model of the stored learning data.

Images