Data processor, method for processing data, program, and data processing system

Abstract

To display an image and a virtual object deposited on each other.SOLUTION: The present invention includes: a first acquisition unit for acquiring image information of a first object; a second acquisition unit for acquiring motion information on motions of a second object different from the first object; a reception unit for receiving data generated by an operation of an external device; and a processing unit for displaying a first image based on the image information, a first virtual object which changes on the basis of the motion information of the second object, and a second image based on the data received by the reception unit, in a display surface.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a data processing device, a data processing method, a program, and a data processing system. [Background technology]

[0002] There is known a technique for displaying a composite image in which a virtual object is superimposed on an image in real space (Patent Document 1, etc.). In the conventional technique, the composite image may look unnatural. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-184689 A Summary of the Invention

[0004] According to one aspect of the present invention, there is provided a data processing device comprising: a first acquisition unit that acquires image information of a first subject; a second acquisition unit that acquires movement information of a second subject different from the first subject; and a processing unit that displays on a display surface an image based on the image information acquired by the first acquisition unit and a virtual object that changes based on the movement information acquired by the second acquisition unit.

[0005] According to another aspect of the present invention, there is provided a data processing method for acquiring image information of a first subject from a first acquisition unit, acquiring movement information of a second subject different from the first subject from a second acquisition unit, and displaying on a display surface an image based on the image information acquired by the first acquisition unit and a virtual object that changes based on the movement information acquired by the second acquisition unit.

[0006] According to another aspect of the present invention, there is provided a program that causes a data processing device to function as a processing unit that displays on a display surface a virtual object that changes based on an image of image information of a first subject acquired from a first acquisition unit and motion information of a second subject different from the first subject acquired from a second acquisition unit. The program is distributed via a portable recording medium or a network, and is installed in a computer such as a data processing device or a server.

[0007] According to another aspect of the present invention, there is provided a data processing system comprising a first acquisition unit that acquires image information of a first subject, a second acquisition unit that acquires movement information of a second subject different from the first subject, and a processing unit that delivers to a terminal device an image based on the image information acquired by the first acquisition unit and a virtual object that changes based on the movement information acquired by the second acquisition unit. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a distribution system. [Diagram 2] FIG. 2 is a block diagram of a user terminal device. [Diagram 3] Block diagram of the server. [Figure 4] 13 is a flowchart of a distribution preparation process. [Diagram 5] 13 is a flowchart of a distribution process when a user is standing still. [Figure 6] (a) is a diagram defining the orientation of an avatar object, (b) is a transition diagram when the avatar object rotates 90 degrees to the right from a state facing away, and (c) is a transition diagram when the avatar object rotates 270 degrees to the left from a state facing away. [Figure 7] 13A and 13B are diagrams showing changes in acceleration when a user moves from the front to the rear. [Figure 8] 13 is a flowchart of a distribution process when a user moves by walking or running. [Figure 9]1A is a diagram showing an example of adding an effect object in addition to an avatar object when a specific motion is detected, and FIG. 1B is a diagram showing an example of adding an effect object. [Figure 10] FIG. 1 is a block diagram of a user terminal device having depth sensors on both its outer and inner surfaces. [Figure 11] (a) is a schematic diagram when a 360-degree camera is used, (b) is a schematic diagram when two hemispherical cameras are used, (c) is a schematic diagram when one hemispherical camera is used, and (d) is a schematic diagram when the image receiving element of the outer camera and the inner camera are shared and used as a single camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A distribution system according to an embodiment of the present invention will be described below with reference to FIGS. [Outline of the distribution system] 1, the distribution system 1 includes a user terminal device 10, a server 30, and a viewer terminal device 50. Note that there may be multiple user terminal devices 10 and multiple viewer terminal devices 50, each of which is connectable to the server 30.

[0010] The user terminal device 10 is a data processing device equipped with multiple cameras. Examples of the user terminal device 10 include a smartphone and a tablet. A user is a person who operates the user terminal device 10, a person who takes pictures using the camera of the user terminal device 10, and a distributor who distributes images using the user terminal device 10.

[0011] The user terminal device 10 has an outer surface facing away from the user holding the user terminal device 10, and an inner surface facing the user. The user terminal device 10 has an outer camera 11 on the outer surface and an inner camera 13 on the inner surface. As an example, the outer camera 11 captures an image of a scene in the line of sight of the user, which is a first direction, as a subject (first subject). Therefore, the subject captured by the outer camera 11 does not include the user.

[0012] On the other hand, as an example, the in-camera 13 captures an image of a subject in a second direction, which is the opposite direction to the above-mentioned first direction. That is, the in-camera 13 captures an image of a subject (second subject) in the direction of the user. Therefore, the subject captured by the in-camera 13 includes the user. The user terminal device 10 can be attached with a support member 3 having an operation unit 22. By attaching the support member 3 to the user terminal device 10, the out-camera 11 can capture an image of a subject in the first direction from a position farther away from the user. Also, the in-camera 13 can capture an image of a subject in the second direction from a position farther away from the user.

[0013] At least one of the outer camera 11 and the inner camera 13 may be an external camera connected to the user terminal device 10 by wire or wirelessly. The user terminal device 10 includes a depth sensor 15. The depth sensor 15 is disposed on the inner surface on which the in-camera 13 is disposed. The depth sensor 15 is disposed near the in-camera 13. As an example, the depth sensor 15 detects the distance between the depth sensor 15 and the user, with the user direction being the second direction as the detection range, similar to the in-camera 13. As an example, the depth sensor 15 detects the user's head, face, torso, hands, arms, legs, etc. As an example, the depth sensor 15 detects the user's movement in the detection range, and generates and outputs motion information based on the user's movement. The depth sensor 15 generates motion information at the same time as the image 101 captured by the out-camera 11. That is, the motion information is synchronized with the image 101, as an example.

[0014] The user terminal device 10 includes a sensor 17. Examples of the sensor 17 include an acceleration sensor and a gyro sensor. The user terminal device 10 detects the moving direction and inclination of the user terminal device 10 by the sensor 17 as a position information detection unit. This allows the user terminal device 10 to calculate position information of the user terminal device 10. The user terminal device 10 also functions as a pedometer by using the data of acceleration, angle, and angular velocity detected by the sensor 17. The pedometer can be realized by installing an application program in the user terminal device 10. The user terminal device 10 includes a GNSS function and can identify the current location.

[0015] The user terminal device 10 includes a display unit 19 having a display surface on which an image is displayed, and a touch panel 20 provided on the display surface. The display unit 19 and the touch panel 20 are disposed on the inner surface of the user terminal device 10 on which the in-camera 13 is disposed. The display surface of the display unit 19 displays an image of a subject captured by the out-camera 11 and the in-camera 13. As an example, an image of a subject captured by the out-camera 11 and a virtual object based on motion information of the subject captured by the in-camera 13 are displayed. Specifically, a composite image is displayed in which a virtual object based on motion information of a user detected by the in-camera 13 is superimposed on the image of a subject captured by the out-camera 11. The virtual object is, for example, an avatar object 102. On the display surface, the display position of the avatar object 102 corresponds to the position where the user is captured by the in-camera 13. Also, the display position of the avatar object 102 corresponds to the position of the user relative to the depth sensor 15. The display position of avatar object 102 is a position specified on touch panel 20 by touching a predetermined position on the display surface, and avatar object 102 is displayed at the specified position on the display surface.

[0016] The avatar object 102 is an incarnation of the user who is operating the user terminal device 10. Examples of the avatar object 102 include a person object imitating a famous person, a famous person, an ordinary person, etc., an animal object imitating an animal, a character object appearing in an animation, an original object created by the user, etc.

[0017] The avatar object 102 changes according to the user's motion information detected by the depth sensor 15. The avatar object 102 also changes according to the detection result by the sensor 17. As an example, when the user detected by the depth sensor 15 moves his / her hand, the avatar object 102 also moves in the same manner according to the motion information of the subject's hand detected by the depth sensor 15. As an example, when the user is walking, the avatar object 102 also moves as if walking according to the detection result of the sensor 17, and when the user is running, the avatar object 102 also moves as if running. That is, when the user moves from place to place by walking or running, the movement direction and movement speed are detected according to the sensor 17 and GNSS in addition to the user's motion information detected by the depth sensor 15, and the avatar object 102 moves as if walking or running.

[0018] The user terminal device 10 is a communication device having a communication function for communicating with the server 30. The user can distribute synthetic data 100 of a synthetic image in which an avatar object 102 is superimposed as a virtual object on an image 101 of a current location, such as a tourist spot, a famous place, a historical site, or a landscape, captured by the rear camera 11. The synthetic data 100 generated by the user terminal device 10 is data for distribution that is transmitted to the server 30 and distributed to the viewer terminal device 50. When the user terminal device 10 transmits such synthetic data 100 to the server 30, the server 30 distributes the synthetic data 100 live. The viewer terminal device 50 can receive and view the synthetic data 100 distributed by the server 30 in real time. The server 30 also has a program guide and can distribute the data at a specified time period.

[0019] The viewer terminal device 50 is a data processing device such as a smartphone or a tablet. The viewer terminal device 50 has at least a communication function capable of communicating with the server 30 and a display unit having a display surface on which an image is displayed. When the server 30 is live-distributing the composite data 100, the viewer terminal device 50 displays a composite image of the composite data 100 on the display surface, and the viewer terminal device 50 can view the composite image in real time. The viewer terminal device 50 can also access the server 30 and check a program guide to view an image of a predetermined composite data 100 at the distribution time. The viewer terminal device 50 can also transmit a request comment to the user terminal device 10 through the server 30 during live distribution, as an example. This allows the user and the viewer to have a conversation. As an example, the user can perform imaging according to a request comment from the viewer.

[0020] [User terminal device 10] As shown in FIG. 2, the user terminal device 10 includes an out-camera 11, an out-camera interface (hereinafter, "interface" will be simply referred to as "IF") 12, an in-camera 13, an in-camera IF 14, a depth sensor 15, a depth sensor IF 16, a sensor 17, and a sensor IF 18. The user terminal device 10 also includes a display unit 19, a touch panel 20, and a communication IF 21. An operation unit 22 of the support member 3 is connected to the communication IF 21. The user terminal device 10 also includes a GNSS receiver 23, an audio input device 24, an audio output device 25, and an audio IF 26. The user terminal device 10 also includes a network IF 27, a storage unit 28, and a processing unit 29.

[0021] The outer camera 11 and the inner camera 13 are equipped with imaging elements such as a CCD sensor, a CMOS sensor, etc. The outer camera 11 is disposed on the outer surface opposite to the user holding the user terminal device 10. The inner camera 13 is disposed on the inner surface facing the user holding the user terminal device 10. A display unit 19 is disposed on the inner surface, and the inner camera 13 is disposed around the display unit 19.

[0022] The outer camera 11 as a first detection unit is connected to the outer camera IF 12 as a first acquisition unit, and outputs imaging data to the outer camera IF 12. The outer camera IF 12 outputs imaging data of a subject captured by the outer camera 11 to a processing unit 29. The inner camera 13 is connected to the inner camera IF 14, and outputs imaging data to the inner camera IF 14. The inner camera IF 14 outputs imaging data of a subject captured by the inner camera 13 to the processing unit 29.

[0023] The outer camera 11 captures an image of a subject such as a scene in the user's line of sight, which is a first direction. The outer camera 11 captures an image of a subject that does not include the user. The inner camera 13 captures an image in the user direction, which is a second direction opposite to the user's line of sight and different from the first direction, to enable a selfie. The subject captured by the inner camera 13 includes the user, but the subject captured by the outer camera 11 does not include the user.

[0024] The outer camera 11 has a first resolution, and the inner camera 13 has a second resolution. The outer camera 11 and the inner camera 13 may have the same resolution, or one of the cameras may have a higher resolution than the other camera. As an example, the outer camera 11, which often captures scenery as a subject, has a higher resolution than the inner camera 13, which often captures a user as a subject.

[0025] Out-camera 11 has a first angle of view, and in-camera 13 has a second angle of view. Out-camera 11 and in-camera 13 may have the same angle of view, or one of the cameras may have a wider angle than the other camera.

[0026] The outer camera 11 and the inner camera 13 are equipped with a zoom lens, and the focal length can be changed by changing the zoom lens in the optical axis direction, thereby enlarging or reducing the captured image. The outer camera 11 and the inner camera 13 may also be equipped with a digital zoom function.

[0027] The outer camera 11 is an imaging unit that captures an image of a scene in the user's line of sight as a subject. A plurality of outer cameras 11 may be arranged. As an example, when two outer cameras 11 are arranged, two images (an image for the right eye and an image for the left eye) for stereoscopic display having parallax can be generated. A plurality of inner cameras 13 may also be arranged. In such a case, when the inner camera 13 is used, two images for stereoscopic display having parallax can be generated.

[0028] Depth sensor 15 is a detection unit and is disposed near in-camera 13. The detection range of depth sensor 15 is approximately the same as the imaging range of in-camera 13. Note that the detection range of depth sensor 15 may be wider or narrower than the imaging range of in-camera 13.

[0029] The depth sensor 15 is an infrared camera, for example. The infrared camera includes a light projecting unit that projects infrared rays and an infrared detection unit that detects infrared rays. The depth sensor 15 acquires depth information such as three-dimensional position information in real space from the time it takes for an infrared pulse projected from the light projecting unit to be reflected and returned. The depth sensor 15 acquires depth information, which is the distance from the depth sensor 15 to the subject, and for example, acquires depth information, which is the distance between the depth sensor 15 and each part of the user (head, upper body, arms, hands, lower body, feet, etc.). The depth sensor 15 then generates movement information according to changes in the depth information of each part. The depth sensor 15 is connected to a depth sensor IF16 as a second acquisition unit, and outputs the movement information to the depth sensor IF16. The depth sensor IF16 outputs the movement information of the subject captured by the depth sensor 15 to a processing unit 29. The depth sensor 15 can obtain depth information of various parts of the user's body without the user having to attach motion sensors to various parts of the body.

[0030] Specifically, the depth sensor 15 extracts the human region of the user and divides it into a human region and a non-human region, for example. As an example, an area is extracted based on the amount of infrared light, and a shape that can be recognized as a human body is determined to be a human region. As an example, the depth sensor 15 acquires 25 skeletal positions of each user as skeletal data, and further calculates depth information for each skeletal position. Examples of the skeletal positions include skeletal positions of the left and right hands, head, neck, left and right shoulders, left and right elbows, left and right knees, left and right feet, etc. The number of skeletal positions to be acquired is preferably 25, but is not limited to this. In addition, if the human region is outside the detection range of the depth sensor 15, the number of skeletal positions will be less than 25. In addition, if the number of skeletal positions is more than 25, the movement can be reproduced more accurately.

[0031] Here, the depth information is, for example, the distance from the objective lens or sensor surface in front of the depth sensor 15 to the skeletal position of the user. The skeletal position is set at the center position of each part by acquiring depth information of each part in the person area, identifying the real space parts of the person reflected in the person area (left and right hands, head, neck, left and right shoulders, left and right elbows, left and right knees, left and right feet, etc.) based on the feature amount of the depth and shape. In addition, the skeletal position is identified by using a feature amount dictionary stored in the storage unit to match the feature amount determined from the person area with the feature amount of each part registered in the feature amount dictionary. In addition, the depth sensor 15 detects the unevenness of the person (shape of the palm, unevenness such as wrinkles in clothes, unevenness of hairstyle, etc.). This enables the depth sensor 15 to track each part and generate movement information according to the change in the depth information of each part (skeleton and movement of each skeleton = movement information). The motion capture process as described above is performed without attaching a marker to the user, and movement information can be easily acquired.

[0032] In addition, the depth information may be calculated by reading a projected infrared pattern and obtaining the depth information from the distortion of the pattern (Light Coding method). Furthermore, the depth information may be calculated from parallax information obtained by a twin-lens camera or a plurality of cameras. Furthermore, the depth information may be calculated by performing image recognition and image analysis on the image acquired by the in-camera 13. In this case, the in-camera 13 functions as a detection unit and the in-camera IF 14 functions as a second acquisition unit, so that the depth sensor 15 and the depth sensor IF 16 are not required. When detecting the skeletal position from the image of the subject captured by the in-camera 13, the skeletal position can be detected by using AI using deep learning. The detection and recognition process of the user's motion (the movement of the skeleton and each skeleton = action) in the motion capture process can also be performed by subjecting the image of the subject captured by the in-camera 13 to recognition processing based on the learning result by deep learning or the like.

[0033] The sensor 17 is, for example, an acceleration sensor or a gyro sensor, and is an acceleration detection unit. The sensor 17 also functions as a step detection unit in conjunction with a pedometer program. The sensor 17 generates detection data such as acceleration data, angle data, and angular velocity data of the user terminal device 10 according to outputs from various sensor elements. The sensor 17 is connected to the sensor IF 18 and outputs the detection data to the sensor IF 18. The sensor IF 18 outputs the detection data detected by the sensor 17 to the processing unit 29. The user terminal device 10 is carried and used by the user. Therefore, the sensor 17 calculates detection data that becomes movement information of the user terminal device 10 accompanying the movement of the user walking or running, and outputs the detection data to the processing unit 29. In addition, even when the user is stationary, the acceleration data, angle data, angular velocity data, etc. of the user terminal device 10 are detected, and detection data serving as posture information of the user terminal device 10 (information on whether the user terminal device 10 is facing upward, downward, rightward or leftward, whether it has moved up and down, forward and backward, or left and right, etc.) is generated and output to the processing unit 29.

[0034] The display unit 19 is disposed on the inner surface of the user terminal device 10 located on the side of the user holding the user terminal device 10. The display unit 19 is disposed on most of the inner surface. The display unit 19 is, for example, a flat display panel such as a liquid crystal display panel or an organic EL panel. The display unit 19 has a display surface. The display surface is provided with a touch panel 20 as a position input unit for inputting a position on the display surface. In the touch panel 20, the input position may be specified by the user's finger or may be specified by a stylus pen. The touch panel 20 may be of a capacitance type, a resistive film type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, or the like. The capacitance type may be of a surface type or a projection type. In the projection type, multi-input by multi-touch is possible, and as an example, the size of the avatar object 102 can be adjusted by opening and closing two fingers (for example, a thumb and an index finger).

[0035] An image or video of a subject captured by the outer camera 11 or the inner camera 13 is displayed on the display surface. An avatar object 102 is displayed superimposed on an image 101 of the subject captured by the outer camera. The position at which the avatar object 102 is displayed can be specified by the touch panel 20. The avatar object 102 moves on the display surface in accordance with information on the user's movement detected by the depth sensor 15, etc.

[0036] The communication IF 21 is a wired or wireless communication unit that communicates with other devices, and examples of the communication IF are USBIF and Bluetooth (registered trademark). The user terminal device 10 can be fitted with a support member 3. The support member 3 is provided with an operation unit 22. The operation unit 22 is connected to the user terminal device 10 via the communication IF 21, and is capable of inputting operations. The operation unit 22 is composed of, for example, a plurality of push buttons, a slide switch, a rotary switch, a cross key, a joystick, a trackball, and the like. For example, when an image of a subject is being captured using the outer camera 11 and / or the inner camera 13, operations such as start recording, stop recording, pause, start playback, stop playback, start distribution, stop distribution, and zoom can be performed from the operation unit 22.

[0037] It should be noted that other external devices can also be connected to the communication IF 21. For example, a camera that replaces the outer camera 11, the inner camera 13, or the depth sensor 15 is connected. In this case, the communication IF 21 functions as an acquisition unit such as a first acquisition unit that acquires the image 101 and a second acquisition unit that acquires the motion information.

[0038] The GNSS receiver 23 detects the current position of the user terminal device 10 by receiving a signal from a GNSS (Global Navigation Satellite System). One example of a GNSS is the Global Positioning System (GPS). Such a positioning system makes it possible to identify the current position of the user terminal device 10 and to grasp the movement history. Furthermore, it is possible to calculate the speed at which the user is moving.

[0039] The audio input device 24 is an acoustic-electrical conversion element such as a microphone, and is connected to the audio IF 26 as an acoustic acquisition unit. The audio input device 24 may be a device built into the user terminal device 10, or may be an external device. If it is an external device, it is connected to the audio IF 26 by wire or wirelessly. When an image of a subject is captured using the outer camera 11 and / or the inner camera 13, the audio input device 24 simultaneously collects sound at the imaging location.

[0040] The audio output device 25 is an electroacoustic conversion device such as a speaker, an earphone, or a headphone, and is connected to the audio IF 26. The speaker of the audio output device 25 may be a device built into the user terminal device 10 or an external device. If it is an external device, it is connected to the audio IF 26 by wire or wirelessly. The audio output device 25 emits an acoustic signal in synchronization with the video being played. In addition, it emits an acoustic signal based on the voice uttered by the user during imaging and surrounding environmental sounds.

[0041] The network IF 27 is a communication unit that communicates with the server 30 via a network 2 such as a wide area network or a local area network. The network IF 27 communicates with the server 30 via a mobile communication network such as a wireless LAN (Wi-Fi) using the IEEE802.11 standard, 3G, 4G / LTE, or 5G. The user terminal device 10 transmits the composite data 100 to the server 30 and can distribute the composite data 100 through the server 30 live or at a predetermined time.

[0042] The storage unit 28 is a storage medium such as a non-volatile memory, and examples of such a storage medium include a hard disk and a flash memory. The storage unit 28 may be configured as a storage medium built into the user terminal device 10, or as a removable storage medium that is detachable from the user terminal device 10. The processing unit 29 is configured with a CPU, RAM, ROM, etc., and controls the overall operation of the user terminal device 10 according to a program.

[0043] A delivery program 28a for delivering an image of a subject captured by the outer camera 11 is installed in the storage unit 28. The storage unit 28 also stores images and videos of a subject captured by the outer camera 11 and the inner camera 13. The storage unit 28 also stores three-dimensional data of an avatar object 102 to be superimposed on the image 101 in association with the delivery program 28a. The avatar object 102 has nodes set that correspond to the skeleton positions detected by the depth sensor 15.

[0044] The distribution program 28a has a function of acquiring user movement information, detection data from the sensor 17, GNSS position information, and sound at the scene where the image 101 is captured by the outer camera 11 at the same time as the image 101 is captured by the outer camera 11, using the depth sensor 15. The distribution program 28a also has a function of generating composite data 100 in which an avatar object 102 is superimposed on the image 101 captured by the outer camera 11 and the avatar object 102 is changed according to information and data such as user movement information. The distribution program 28a also has a function of transmitting the composite data 100 to the server 30 while displaying the composite data 100 on the display surface of the display unit 19. The distribution program 28a also has a function of viewing the composite data 100 on the viewer terminal device 50. The processing unit 29 executes the functions of the distribution program 28a in accordance with a user's operation.

[0045] In addition, the distribution program 28a installed in the user terminal device 10 uses a functional portion for distribution, and the distribution program 28a installed in the viewer terminal device 50 uses a functional portion for viewing and a functional portion for sending comments to the user terminal device 10.

[0046] Furthermore, the storage unit 28 is installed with a pedometer program 28b. The pedometer program 28b has thresholds, such as a first threshold for determining whether the user is standing still or walking and a second threshold for determining whether the user is walking or running, at least with respect to acceleration. The pedometer program 28b can determine whether the user is walking or standing still using the first threshold, and can determine whether the user is walking or running using the second threshold. When the pedometer program 28b is started, the processing unit 29 detects up-down and forward-backward movements based on the acceleration, angle, and angular velocity detected by the sensor 17, and counts the number of steps based on the results. The pedometer may be external to the user terminal device 10. The angle and angular velocity may also have first and second thresholds.

[0047] As an example, the delivery program 28a and the pedometer program 28b are linked. For example, the pedometer program 28b runs in the background of the delivery program 28a. By linking the delivery program 28a and the pedometer program 28b, the movement of the user and the movement of the user terminal device 10 can be detected separately. As an example, when a user holding the user terminal device 10 moves the user terminal device 10 forward, backward, left and right while remaining stationary, the pedometer program 28b does not count the number of steps even if the sensor 17 detects acceleration, etc. In contrast, when a user holding the user terminal device 10 moves by walking or running, the sensor 17 detects acceleration, etc., and the pedometer program 28b counts the number of steps when the acceleration exceeds a first threshold. The processing unit 29 performs a process of superimposing an avatar object 102 that changes according to the user's movement information on an image 101 captured by the outer camera 11 in accordance with the delivery program 28a, and generating composite data 100. At this time, when the processing unit 29 detects that the user is walking or running according to the pedometer program 28b, it generates the synthetic data 100 in which the avatar object 102 also changes to appear as if it is walking or running.

[0048] Furthermore, the storage unit 28 is installed with a voice recognition program 28c. The voice recognition program 28c performs voice recognition processing on the user's voice data collected by the audio input device 24 to generate text data. As an example, the voice recognition program 28c identifies phonemes from a voice waveform, matches the sequence of phonemes with a pre-registered dictionary, converts the phonemes into words, and generates text data of the converted sentence. As an example, the voice recognition program 28c also works in conjunction with the distribution program 28a. When the processing unit 29 performs processing to superimpose an avatar object 102 on an image 101 captured by the outer camera 11 and generate composite data 100 in accordance with the distribution program 28a, the processing unit 29 performs voice recognition processing to generate text data and add an effect object to the image 101 in parallel.

[0049] [Viewer terminal device 50] The viewer terminal device 50 is a data processing device having a configuration similar to that of the user terminal device 10. In the following, the configuration of each part of the viewer terminal device 50 is assigned the same reference numerals as those of the user terminal device 10, and details thereof will be omitted. The viewer terminal device 50 is a terminal for viewing composite data 100, which is generated by superimposing an avatar object 102 that changes according to the user's movement information on an image 101 captured by the out-camera 11 by the user and distributed by the user terminal device 10 through the server 30. The composite data 100 may be generated by the viewer terminal device 50 or may be generated by the server 30 described below. Therefore, the viewer terminal device 50 is required to have at least a function as a communication device and a function as a display device. In addition, the viewer terminal device 50 can also be used as the user terminal device 10 as long as it has a configuration similar to that of the user terminal device 10.

[0050] [Server 30] The server 30 provides information and processing results in response to requests from the user terminal devices 10 and viewer terminal devices 50 which serve as clients.

[0051] As shown in FIG. 3, the server 30 is a device that manages the distribution system 1, and includes a distribution management database 31, a storage unit 32, a network IF 33, and a control unit . The distribution management database 31 manages a content ID that uniquely identifies a content to be distributed, a distribution time, and a viewer account that uniquely identifies a distribution destination for a viewer, in association with a distributor account that uniquely identifies a user who is the owner of the user terminal device 10, i.e., a distributor. Examples of the distributor account include a distributor ID and a distributor password. Examples of the viewer account include a viewer ID and a viewer password.

[0052] For example, it is managed that the distributor account "AAA" is live-distributing the content ID "A123" through the user terminal device 10. It is also managed that the content ID "A123" is being viewed by the viewer terminal IDs "XXX", "YYY", and "ZZZ". It is also managed that the distributor account "BBB" is distributing the content ID "B456" from 17:00 to 18:00 on December 25, 2018. In this case, the content ID "B456" to be distributed is uploaded from the user terminal device 10 in advance and stored in the storage unit 32. It is also managed that the viewer accounts "XYZ", "PPP", and "QQQ" are viewing the content ID "B456" through the viewer terminal device 50 during the playback distribution time period. In this way, by associating the viewer accounts at the time of distribution, for example, it is also possible to transmit comments transmitted from the viewer terminal device 50 to the user terminal device 10.

[0053] The storage unit 32 is a non-volatile memory, such as a hard disk or flash memory, and stores the delivery management program 32a and the above-mentioned delivery management database 31. The network IF 33 is an IF for communicating with the user terminal device 10 and the viewer terminal device 50 via the network 2. The control unit 34 is composed of a CPU, RAM, ROM, etc., and controls the overall operation of the user terminal device 10 in accordance with a program.

[0054] The control unit 34 is a delivery unit that controls delivery of the composite data 100 transmitted from the user terminal device 10 when the network IF 33 receives a delivery request from the user terminal device 10. For example, when the control unit 34 receives a live delivery request, the control unit 34 makes the composite data 100 transmitted sequentially from the user terminal device 10 available for delivery on a predetermined channel. When the control unit 34 receives a delivery request for delivery at a predetermined time and time period, the control unit 34 registers the channel and the delivery time period in a program guide, and makes the composite data 100 available for delivery at the registered predetermined time and time period. When the control unit 34 receives a viewing request from the viewer terminal device 50, the control unit 34 makes the composite data 100 of the content ID corresponding to the received viewing request available on the viewer terminal device 50. The viewer terminal device 50 performs streaming playback. Of course, the viewer terminal device 50 may download the composite data 100 and store it in the storage unit 28.

[0055] The operation of the distribution system will now be described. [Distribution preparation process] While capturing an image of the current location, such as a tourist spot, a famous place, a historical site, or a landscape, the user prepares to distribute the image 101 at the location via the server 30. First, the user connects the support member 3 to the user terminal device 10 so that the user can capture images in the first direction and the second direction from a position farther away from the user. Note that the support member 3 does not have to be connected.

[0056] The user operates the user terminal device 10 owned by the user to start the delivery program 28a. In conjunction with this, the processing unit 29 of the user terminal device 10 starts the depth sensor 15. The depth sensor 15 identifies each part in the person area by identifying the user's skeletal position. The depth sensor 15 then acquires depth information and the like indicating the distance between the depth sensor 15 and each part of the user (head, upper body, arms, hands, lower body, feet, etc.), becomes capable of tracking each part, and generates movement information according to changes in the depth information of each part.

[0057] 4, in step S1, the processing unit 29 acquires user movement information from the depth sensor 15 through the depth sensor IF 16. That is, the processing unit 29 detects the user with the depth sensor 15, recognizes the user, and becomes capable of tracking the user's movement. In addition, the processing unit 29 may also activate the in-camera 13 to acquire an image or video of the user detected by the depth sensor 15.

[0058] In addition to the user's body movements, facial expressions, i.e., changes in the shape of the eyes and mouth, may be detected. The detection results can be reflected in the facial expressions of the avatar. Furthermore, the user's movements and facial expressions may be detected using not only the depth sensor 15 but also the image from the in-camera 13, or may be detected using information from both sensors in combination.

[0059] In step S2, the processing unit 29 repeatedly determines whether the motion information has been acquired through the depth sensor IF 16, and when the motion information has been acquired, the process proceeds to step S4. In addition, in step S3, the processing unit 29 starts the outer camera 11, and acquires an image 101 in a first direction, which is the user's line of sight, through the outer camera IF 12. This image 101 becomes a background image of the avatar object 102, and is, for example, an image that the user wants to introduce to the viewers through the avatar object 102. This allows the processing unit 29 to acquire motion information from the depth sensor 15 and the image 101 from the outer camera 11 at the same time, and to associate them.

[0060] The processing unit 29 may generate composite data for displaying the image 101 captured by the outer camera 11 on the display unit 19, and superimposing a user image object of the user captured by the inner camera 13 and detected by the depth sensor 15 on the image 101 at the same time, and display the composite data on the display surface of the display unit 19. In this case, the processing unit 29 may, for example, extract a user image object from the image captured by the inner camera 13 according to the detection result from the depth sensor 15. Then, the processing unit 29 causes the user image object to be displayed on the image 101. The position at which the user image object is displayed on the display surface is an initial position that is predetermined on the image 101 or the display surface of the display unit 19 by the delivery program 28a. The avatar object 102 is a virtual object that is an incarnation of the user, and the user can predict at which position on the display surface the avatar object 102 will be displayed through the user image object.

[0061] Note that a temporary avatar object may be displayed in the initial position instead of the user image object until the avatar object 102 is officially determined. Also, the process of displaying such a user image object or a temporary avatar object in the image 101 may be omitted.

[0062] The processing unit 29 displays a selection screen for the avatar object 102 on the display surface of the display unit 19. As an example, a list of a plurality of avatar objects 102 stored in the storage unit 28 in association with the distribution program 28a is displayed. If it is not possible to display all the avatar objects at once on the display surface, the list display is scrolled up and down or left and right so that the user can check all the avatar objects that are candidates for selection. The user touches a desired avatar object using a finger or a stylus pen. In step S4, the processing unit 29 selects an avatar object at the touched position among the plurality of avatar objects displayed in the list, specifies the selected avatar object, and determines the avatar object 102 to be used. It is to be noted that further candidate avatar objects may be downloaded from the server 30.

[0063] In step S5, processing unit 29 judges whether a display position for displaying avatar object 102 has been specified within a predetermined period of time. Specifically, processing unit 29 judges whether a predetermined position on the display surface of display unit 19 has been touched by a user using a finger, a stylus pen, or the like within the predetermined period of time. Then, when touched, processing unit 29 designates the touched position as the position for displaying avatar object 102. On the other hand, when not touched, in step S5-1, an initial position predetermined by distribution program 28a on image 101 or the display surface of display unit 19 is set as the position for displaying avatar object 102.

[0064] In step S6, processing unit 29 displays background image 101 on the display surface, generates composite data 100 by superimposing avatar object 102 on image 101 so that avatar object 102 is displayed at a predetermined display position on the display surface, and displays the composite image on the display surface of display unit 19. Specifically, when a user image object of a user captured by in-camera 13 is displayed on image 101, processing unit 29 generates composite data 100 by superimposing avatar object 102 in place of the user image object or the temporary avatar object at the position where the user image object or the temporary avatar object is displayed. Then, processing unit 29 displays the composite image based on the composite data 100 on the display surface of display unit 19.

[0065] In this way, in image 101, avatar object 102 is not superimposed on the user object captured by in-camera 13. This makes it possible to prevent avatar object 102 from being displayed protruding from the user object. As an example, avatar object 102 smaller than the user image object or the temporary avatar object may be displayed.

[0066] Incidentally, avatar object 102 is an object occupying a predetermined area within the display surface on which image 101 is displayed. Therefore, as an example, when a point is specified as the display position of avatar object 102, processing unit 29 displays avatar object 102 on the display surface so that the specified position is the center of avatar object 102. As another example, when the display position of avatar object 102 is specified to have an area using a finger, a stylus pen, or the like, processing unit 29 calculates the center of the area and displays avatar object 102 on the display surface of display unit 19 so that the center coincides with the center of avatar object 102.

[0067] The display portion of the avatar object 102 corresponds to the range detected by the depth sensor 15 of the user, for example. That is, when the depth sensor 15 detects only the upper body of the user, the range displayed by the avatar object 102 is also the upper body. Also, when the depth sensor 15 detects the entire body of the user, the range displayed by the avatar object 102 is also the entire body. Also, the display portion of the avatar object 102 may not correspond to the range detected by the depth sensor 15 of the user. In this case, even if the depth sensor 15 detects only the upper body of the user, the avatar object 102 displays the entire body (from head to toe). Or, even if the depth sensor 15 detects the entire body of the user, the avatar object 102 displays the upper body. Furthermore, the part not detected by the depth sensor 15 may be stationary, but can also be changed according to the detection result of the sensor 17. Or, the movement of the part not detected by the depth sensor 15 can be used to infer and change the movement of the part not detected.

[0068] However, the size of the avatar object 102 in the image 101 displayed on the display surface of the display unit 19 may be too large or too small. Also, the user may want to adjust the size of the avatar object 102 to suit his or her own taste. To handle such cases, in step S7, the processing unit 29 displays a size determination button on the display surface. Then, the processing unit 29 determines whether the size determination button has been touched, and in step S8, executes a process of adjusting the size of the avatar object 102 by the user until the determination button is touched.

[0069] As an example, when the support member 3 is used, the size of the avatar object 102 in the image 101 displayed on the display surface can be changed by changing the distance between the user and the depth sensor 15 of the user terminal device 10. Specifically, by using the support member 3 to move the user away from the depth sensor 15 of the user terminal device 10, the avatar object 102 can be made smaller in the image 101 displayed on the display surface. Also, by bringing the user closer to the depth sensor 15 of the user terminal device 10, the avatar object 102 can be made larger in the image 101 displayed on the display surface. Also, the size of the avatar object 102 can be changed by using the zoom function of the in-camera 13. For example, when the focal length of the in-camera 13 is changed in the wide-angle direction, the avatar object 102 is reduced, and when the focal length is changed in the telephoto direction, the avatar object 102 is enlarged.

[0070] Furthermore, if the touch panel 20 is of the capacitive and projection type, multi-touch is possible, and the size of the avatar object 102 can be adjusted by spreading or closing two fingers. For example, when two fingers are spread apart, the avatar object 102 can be made larger according to the amount by which the two fingers are spread apart, and when two fingers are closed, the avatar object 102 can be made smaller according to the amount by which the two fingers are closed.

[0071] When the size determination button is touched in step S7, processing unit 29 proceeds to step S9. When using support member 3 to change the distance between the user and depth sensor 15 of user terminal device 10, the same determination process as that performed when the size determination button is touched on operation unit 22 of support member 3 can be performed. As a result, avatar object 102 is superimposed on image 101 at the size of avatar object 102 at the time of the determination process.

[0072] After the size of the avatar object 102 is adjusted, the orientation of the avatar object 102 is set. The user is in a state where the face faces the depth sensor 15 of the user terminal device 10, and faces the depth sensor 15. Therefore, the avatar object 102 is also in a state where the face faces forward, that is, the face is reflected. On the other hand, when the user is moving forward, since the avatar object 102 is an incarnation of the user, it is also preferable that the avatar object 102 is in a state where it is walking facing the same direction as the user, that is, the back of the avatar object 102 is displayed with the back of the head and back of the avatar object 102 displayed. Furthermore, when the avatar object 102 is orally introducing famous places, historical sites, etc. reflected in the background image 101 on behalf of the user, it is also preferable that the avatar object 102 is also in a state where the face faces forward, that is, the face is reflected. Depending on the imaging situation, it may be preferable that the avatar object 102 faces sideways regardless of the user's orientation.

[0073] Therefore, processing unit 29 displays a direction determination button on the display surface. Then, in step S9, processing unit 29 determines whether the direction determination button has been touched, and in step S10, executes a process of adjusting the direction of avatar object 102 by the user until the direction determination button is touched.

[0074] As an example, when the support member 3 is used, the orientation of the avatar object 102 can be changed by the operation unit 22 of the support member 3. For example, the orientation of the avatar object 102 can be easily changed by operating the operation unit 22 with a cross key, joystick, trackball, or the like in the direction in which the avatar object 102 is to be faced. Also, the orientation of the avatar object 102 can be changed by using the touch panel 20. As an example, when the touch panel 20 is a capacitive type and a projection type, the orientation of the avatar object 102 can be easily changed by the user tracing his / her finger in the direction in which the avatar object 102 is to be faced. The size and orientation of the avatar object 102 set here become the basis at the start of distribution. That is, at the time of distribution, the display state of the set avatar object 102 changes according to the movement information, etc.

[0075] When the orientation decision button is touched in step S9, the processing unit 29 proceeds to step S11. Note that it is also possible to carry out the same decision processing as when the orientation decision button is touched on the operation unit 22 of the support member 3.

[0076] In step S11, the processing unit 29 displays a distribution start button on the display surface. When the distribution start button is touched, distribution starts. Note that the same decision processing as when the distribution start button is touched on the operation unit 22 of the support member 3 can also be performed.

[0077] The processing unit 29 associates the content with the distributor account and content ID, and transmits the distribution date and time to the server 30 via the network IF 27 whether to perform live distribution or to specify the distribution date, time and time zone. The control unit 34 of the server 30 then registers this in the distribution management database 31.

[0078] When the distribution start process is performed, the processing unit 29 generates composite data 100 of the image 101 with the avatar object 102 superimposed thereon, and transmits the composite data 100 to the server 30 while displaying the composite data 100 on the display unit 19. Then, the control unit 34 of the server 30 starts distribution of the composite data 100 transmitted from the user terminal device 10 through the network IF 33 using the assigned channel (e.g., URL).

[0079] [Distribution processing example 1] 5, the processing unit 29 acquires user movement information from the depth sensor 15 in step S21. In step S22, the processing unit 29 generates composite data 100 including video and audio in which the avatar object 102 changes in accordance with the movement information, and transmits the composite data 100 to the server 30.

[0080] When the user is talking, the depth sensor 15 detects, for example, the movement of the eyes and lips, and when the user moves the hands, the movement of the hands is detected. Also, when the user turns sideways, the movement of turning sideways is detected. The processing unit 29 changes the avatar object 102 according to the user's movement information from the depth sensor 15. That is, when the user is talking, the lips of the avatar object 102 are changed in the same way as the user, and when the user moves their hands, the hands of the avatar object 102 are changed in the same way as the user's hands. Furthermore, when the user turns sideways, the avatar object 102 is turned sideways. The user's movements may be acquired not only from the depth sensor 15, but also from an image from the in-camera 13.

[0081] The processing unit 29 also displays the composite data 100 being transmitted to the server 30 on the display surface of the display unit 19. Therefore, the user can capture an image for distribution while checking how the avatar object 102 is displayed on his / her behalf (see FIG. 1).

[0082] The viewer terminal device 50 can view a composite image based on the composite data 100 being distributed by accessing a predetermined channel of the server 30. The composite data 100 is data in which the lips of the avatar object 102 move in accordance with the lip movement information of the user. Therefore, the viewer sees as if the avatar object 102 is speaking.

[0083] As shown in Fig. 6(a), the data of the avatar object 102 is three-dimensional data. Therefore, when the avatar object 102 faces backward, the back of the head is displayed, when the avatar object 102 faces forward, the face is displayed, when the avatar object 102 faces right, the right side of the face is displayed, and when the avatar object 102 faces left, the left side of the face is displayed. As an example, the processing unit 29 changes the avatar object 102 based on the motion information from the depth sensor 15 and the detection data from the sensor 17 with respect to the state before the action. As shown in Fig. 6(b), when the avatar object 102 rotates 90 degrees rightward from a state facing backward, the avatar object 102 rotates rightward in order.

[0084] FIG. 6(c) shows a case where the avatar object 102 rotates 270 degrees to the left from a state where it faces the back and faces the right. In this case, the avatar object 102 rotates leftward in order. However, the state after the rotation is the same as the state where the avatar object 102 faces rightward after rotating 90 degrees from a state where it faces the back. In this case, as shown in FIG. 6(a), the processing unit 29 may perform a process of rotating the avatar object 102 in a direction in which the rotation angle of the avatar object 102 becomes smaller, that is, to the right. This can reduce the amount of calculation processing for rotating the avatar object 102 in the processing unit 29. In addition, the movement of the avatar object 102 is reduced, and a more natural way in which the avatar object 102 turns around can be reproduced.

[0085] [Distribution processing example 2] The distribution process example 2 is a case where the user is moving. The sensor 17 is an acceleration sensor or a gyro sensor, and the detection data such as acceleration data, angle data, and angular velocity data of the user terminal device 10 are calculated according to the output from these sensor elements. FIG. 7 shows an example of a change in acceleration detected by the sensor 17 when the user moves from the front to the back. At time t1, the user is stationary and the acceleration is 0. When the user starts walking in the back direction at time t2, a positive acceleration is output, and when the user starts walking at a constant speed at time t3, the acceleration approaches 0. When the user stops walking, a negative acceleration is output at time t4. When the user resumes walking in the back direction at time t5, a positive acceleration is output, and when the user is walking at a constant speed at time t6, the acceleration approaches 0. When the user starts running in the back direction at time t7, a positive acceleration higher than when the user started walking is output, and when the user is running at a constant speed as at time t8, the acceleration approaches 0.

[0086] As described above, when the user moves by walking or running, the processing unit 29 of the user terminal device 10 can detect the moving direction and moving speed based on the acceleration, angle, and angular velocity detected by the sensor 17. As shown in Fig. 8, in step S31, the processing unit 29 determines whether the acceleration has changed, and in step S32, changes the avatar object 102 according to the acceleration. When walking or running, up-down and forward-backward movements occur, and when running, the changes are faster and larger than when walking.

[0087] Specifically, the processing unit 29 operates the pedometer program 28b linked to the delivery program 28a. The processing unit 29 then determines whether the acceleration is greater than a first threshold, and if it is less than the first threshold, determines that the person is stationary, and if it is equal to or greater than the first threshold, determines that the person is walking. The processing unit 29 also determines whether the acceleration is greater than a second threshold, and if it is less than the second threshold, determines that the person is walking, and if it is equal to or greater than the second threshold, determines that the person is running.

[0088] The processing unit 29 changes the avatar object 102 up and down and forward and backward when walking or running, and when running, the amount of change is greater than when walking. When the entire body of the avatar object 102 is displayed on the display screen, the processing unit 29 changes the arms and legs as if the avatar object 102 is running or walking, even if the depth sensor 15 does not detect the arms and legs. Even when only the upper body of the avatar object 102 is displayed on the display screen, the processing unit 29 changes the avatar object 102 so that it moves up and down.

[0089] In the example of FIG. 7, the user is moving forward from the front to the back. In reality, the depth sensor 15 of the user terminal device 10 faces the user's face, but the processing unit 29 displays the back of the head and the back of the avatar object 102 as if the avatar object 102 is moving from the front to the back. When the user is stationary, such as from time t4 to time t5, the processing unit 29 reverses the orientation of the avatar object 102 so that the face faces forward. When talking about famous places, historical sites, etc., the user is often stationary, and when explaining, it is easier for the viewer to understand the user's facial expressions through the facial expressions reproduced in the avatar object 102 if the face of the avatar object 102 is visible. In the above example, when the user is moving, the back of the avatar object 102 is displayed, but the avatar object 102 may always face forward.

[0090] The movement of the user may be detected not only by the sensor 17 but also by a combination of other means. For example, the sensor 17 may be used in combination with GNSS. Furthermore, the depth sensor 15 and SLMA (Simultaneous Localization and Mapping) may be used in combination. Furthermore, the change over time in position information obtained by photogrammetry (VisualSLAM) may be used supplementarily to determine the walking direction.

[0091] [Other distribution processing examples] Furthermore, when the processing unit 29 detects a specific motion based on the motion information from the depth sensor 15, it additionally displays an effect object 103 related to the motion in addition to the avatar object 102. As an example, when a person hits the palm of one hand with the fist of the other hand, it is a flash of inspiration. Therefore, as shown in FIG. 9(a), when the processing unit 29 detects a motion in which a user hits the palm of one hand with the fist of the other hand based on the motion information from the depth sensor 15, it displays an effect object 103 as an additional virtual object resembling a light bulb near the avatar object 102. The effect object 103 may be a stationary object or a moving object, and may further be audio data such as a voice or music that is reminiscent of a flash of inspiration.

[0092] 9(b), while a process of overlaying an avatar object 102 on an image 101 captured by the outer camera 11 to generate composite data 100 is being performed in accordance with the distribution program 28a, a process of performing voice recognition to generate text data and adding an effect object 104 to the image 101 is performed in parallel. The effect object 104 here can be configured by adding a speech bubble object to the avatar object 102 and displaying text in the speech bubble object. Alternatively, the effect object 104 may be configured by displaying text in a column object at the top, bottom, left, or right in a composite image based on the composite data 100.

[0093] Furthermore, when the processing unit 29 detects a specific action based on the movement information from the depth sensor 15, it may add a speech bubble object to the avatar object 102 and display text in the speech bubble object to configure the effect object 104. In this case, the specific action is, for example, a specific gesture when introducing something, such as raising one palm up to shoulder height.

[0094] Furthermore, a viewer watching on the viewer terminal device 50 can send a comment to the user who is broadcasting. In the server 30, the broadcast management database 31 manages the viewer account in association with the broadcaster account. When the control unit 34 of the server 30 receives comment data associated with the broadcaster account or the content ID from the viewer terminal device 50, the control unit 34 transmits the comment to the user terminal device 10 of the corresponding broadcaster. In such a case, the user terminal device 10 displays the comment from the viewer on the display screen of the display unit 19. For example, if a user is broadcasting while walking around a tourist spot, the viewer can send a comment requesting the next place he or she wants the user to go to the user terminal device 10 from the viewer terminal device 50 via the server 30. In this way, the viewer can send a comment to the user, so the user can shoot while communicating with the viewer.

[0095] Furthermore, the viewer can display a virtual object related to the viewer himself / herself in the image 101 through the viewer terminal device 50. For example, the viewer's own avatar object can be added to a composite image based on the composite data 100. The avatar object, which is a virtual object of the viewer, is changed based on the movement information generated by the depth sensor 15 included in the viewer terminal device 50, and is transmitted from the viewer terminal device 50 to the server 30, and is further displayed on the user terminal device 10 via the server 30. In this case, the composite data including the avatar object 102 of the distributor and the avatar object of the viewer may be generated by any of the viewer terminal device 50, the server 30, and the user terminal device 10. Then, the avatar object 102 of the distributor and the avatar object of the viewer are displayed in the composite image based on the composite data 100. As for the avatar object of the viewer, an avatar object to be displayed in the image 101 is selected by the viewer terminal device 50, and the selected avatar object is displayed at a predetermined position in the image 101. Furthermore, instead of an avatar object, a virtual object such as a present for the avatar object 102 may be displayed. For example, virtual objects include an offering, a microphone for broadcasting, a bouquet of flowers, glasses, clothing, and other accessories. The viewer can select such a present from a present list screen. The selected virtual object is then transmitted to the server 30, According to the above-described distribution system 1, the following effects can be obtained.

[0096] (1) The avatar object 102 is not superimposed on the user object of the user, but is superimposed on the image 101 in place of the user object. This prevents the movement of the avatar object 102 from being unable to follow the movement of the user object, and prevents the user object from protruding from the avatar object 102. Also, when the user object is smaller than the avatar object 102, it prevents the user object from protruding from the avatar object 102. In this way, it is possible to prevent the avatar object 102 from appearing unnatural in the composite image.

[0097] (2) The avatar object 102 can be automatically added to the image 101. Alternatively, the position at which the avatar object 102 is to be displayed can be easily specified by using the touch panel 20.

[0098] (3) The user terminal device 10 can use the outer camera 11 to capture an image of a background subject to generate an image 101, and the depth sensor 15 can then detect the user who captured the image and add an avatar object 102 to the image 101.

[0099] (4) Since the user terminal device 10 includes the sensor 17, the avatar object 102 can be changed using detection data from the sensor 17 even in areas where the user's movement cannot be detected by the depth sensor 15.

[0100] (5) The user terminal device 10 has a pedometer function. Therefore, it is possible to detect the movement of the user and the movement of the user terminal device 10 separately, that is, it is possible to determine whether the user is moving only the user terminal device 10 with his / her hand while the user is stationary, or whether the user is actually walking. When it is detected that the user is walking, it is also possible to change the avatar object 102 to look like the user is walking. Furthermore, it is possible to detect whether the user is running, and when the user is running, it is possible to change the avatar object 102 to look like the user is running. And, by using the depth sensor 15, the outer camera 11, the sensor 17, and the pedometer function, the distributor can walk ahead of the viewer without walking backwards, and perform as if he / she is a tourist guide.

[0101] (6) When the processing unit 29 detects a specific movement from the movement information, it adds an effect object 103. This makes it possible to diversify the expression style of the avatar object 102 and further increase entertainment value.

[0102] (7) The processing unit 29 recognizes the user's voice, converts it into text, and adds the text to the image 101. This makes it possible for the viewer to easily understand what the user is saying.

[0103] The above-described live distribution system can be modified as follows. The user terminal device 10 may be configured as shown in Fig. 10. That is, the user terminal device 10 shown in Fig. 10 includes an out-depth sensor 41 on the outer surface opposite to the user holding the user terminal device 10, and the out-depth sensor 41 is connected to the out-camera IF42. The out-depth sensor 41 is arranged near the out-camera 11, for example. Note that in Fig. 10, the depth sensor arranged on the inner surface on the user side is the in-depth sensor 15, which is connected to the in-depth sensor IF16. The in-depth sensor 15 and the in-depth sensor IF16 correspond to the depth sensor 15 and the depth sensor IF16 in Fig. 2, respectively.

[0104] The out-depth sensor 41 has a similar configuration to the in-depth sensor 15, and is, for example, an infrared camera, and includes a light-projecting unit that projects infrared rays and an infrared detection unit that detects the infrared rays. Then, depth information such as three-dimensional position information in real space is obtained from the time it takes for an infrared pulse projected from the light-projecting unit to be reflected and returned.

[0105] The in-depth sensor 15 mainly detects a user holding the user terminal device 10 in his / her hand. That is, the in-depth sensor 15 detects objects that are relatively close to the in-depth sensor 15. In contrast, the out-depth sensor 41 detects objects that are farther away compared to the distance to the detection target of the in-depth sensor 15. For this reason, as an example, it is preferable that the light projecting section of the out-depth sensor 41 has a higher output than that of the in-depth sensor 15, and the detection section has a higher sensitivity than that of the in-depth sensor 15.

[0106] The out-depth sensor 41 is disposed near the out-camera 11, and its detection range is approximately the same as the imaging range of the out-camera 11. The detection range of the out-depth sensor 41 may be wider or narrower than the imaging range of the out-camera 11. The out-depth sensor 41 detects depth information of an object present within the detection range. As an example, when a person is walking in the user's traveling direction, the out-depth sensor 41 detects the movement of the person. Also, when there are stairs ahead, the out-depth sensor 41 detects each step of the stairs. Furthermore, when a person ahead is climbing the stairs, the out-depth sensor 41 detects the movement. Then, the out-camera IF 42 as a depth information acquisition unit acquires the depth information from the out-depth sensor 41.

[0107] When generating the composite data 100 of the image 101 on which the avatar object 102 is superimposed and distributing the composite data 100 through the server 30, the processing unit 29 utilizes the depth information and the motion information from the out-depth sensor 41 as follows. As an example, when a person in front is climbing stairs, the probability that the next user will also climb stairs is high. The out-depth sensor 41 detects the stairs and the motion of the person in front. The processing unit 29 detects that the user has reached the stairs according to the depth information and the motion information from the out-depth sensor 41. When the processing unit 29 detects that the user has transitioned to a state of climbing stairs based on the motion information from the in-depth sensor 15 and the detection data from the sensor 17, the processing unit 29 applies the motion information when the person in front is climbing the stairs to the avatar object 102, and changes the stairs so that the avatar object 102 looks as if it is climbing the stairs. In this way, when stairs exist in the real space, the avatar object 102 can be displayed as if it is climbing the stairs, and it is possible to prevent the avatar object 102 from changing unnaturally with respect to the image 101.

[0108] Here, the case where the user climbs the stairs in accordance with the person in front has been described, but this is just one example. For example, if an obstacle such as a flower bed, a desk, or a chair is present ahead in the moving direction, the user operates to avoid the obstacle. The processing unit 29 can change the avatar object 102 so as to avoid the obstacle based on the depth information of the obstacle detected by the out-depth sensor 41. That is, the avatar object 102 can be displayed in consideration of the front-to-back relationship with real objects such as a flower bed, a desk, or a chair. In this way, the user terminal device 10 equipped with the out-depth sensor 41 can change the avatar object 102 in accordance with the object in the real space imaged by the out-camera 11.

[0109] The user terminal device 10 may use a 360-degree camera device 201 paired by wire or wirelessly through a communication IF 21 serving as a first acquisition unit and a second acquisition unit, instead of using the built-in out-camera 11 and in-camera 13. As shown in FIG. 11(a), the 360-degree camera device 201 includes a first camera 202 facing a first direction on a first surface, and a second camera 203 facing a second direction on a second surface opposite to the first surface. Each of the first camera 202 and the second camera 203 includes an image receiving element. In this case, one of the first camera 202 and the second camera 203 corresponds to the out-camera 11 described above, and the other corresponds to the in-camera 13 described above. In addition, a depth sensor is disposed at least near the lens of the camera corresponding to the in-camera 13. This results in a user terminal device 10 similar to that of FIG. 2, and when depth sensors are disposed near both cameras, the user terminal device 10 becomes similar to that of FIG. 10.

[0110] It should be noted that, in the case where the 360-degree camera device 201 has a configuration similar to that of the user terminal device 10 in FIG. 2 or FIG. 10, the 360-degree camera device 201 itself becomes the user terminal device 10 in FIG.

[0111] A 360-degree camera device 201 with such a configuration is preferably used by attaching it to an attachment device for fixing the camera to a helmet, an attachment device for wearing the camera attached to a wrist mount on the back of the hand, wrist, or arm, an attachment device for fixing the camera by clamping it to the handlebars of a bicycle with a fixing clamp, an attachment device for fixing the camera to the body, an attachment device for fixing the camera with a suction plate, or a handy stick of fixed length.

[0112] As shown in FIG. 11(b), the first camera 206 and the second camera 207 may be separated. That is, two semi-spherical cameras are used. In this case, a depth sensor is disposed at least on the camera corresponding to the in-camera 13. This results in a user terminal device 10 similar to that shown in FIG. 2, and when depth sensors are disposed on both cameras, the user terminal device 10 results in a user terminal device 10 similar to that shown in FIG. 10. Since the first camera 206 and the second camera 207 are separated, the camera corresponding to the out-camera 11 and the camera corresponding to the in-camera 13 do not need to be in the same place. As an example, by connecting the first camera 206 and the second camera 207 to the user terminal device 10 by wire or wirelessly, the user and the camera corresponding to the in-camera 13 do not need to be present at the installation location of the camera corresponding to the out-camera 11. The user can install the camera corresponding to the out-camera 11 in a place remote from the user's own place of residence, such as a home, and live-stream the composite data 100 while staying at the place of residence, such as a home.

[0113] The example of FIG. 11(c) uses one semi-spherical camera 211. As an example, the semi-spherical camera 211 is a wide-angle camera. The semi-spherical camera 211 is paired with the communication IF 21 in a wired or wireless manner. In this case, the angle of view is, for example, about 170 degrees to about 185 degrees, which is a wide angle. An image incident from one lens unit 212 is received by one image receiving element. In the captured image 213, a user appears in a part 214, and the other area 215 is the area that becomes the image 101.

[0114] The hemispherical camera 211 does not have a depth sensor, and the depth information and movement information of the user are calculated by image analysis using photogrammetry technology or the like. The depth information and movement information of the user can be detected by using AI that utilizes deep learning. The processing unit 29 overlays an avatar object 102, which changes according to the user's movement information detected in the area 214, on the image 101 in the area 215 to generate composite data 100. The user terminal device 10 distributes this composite data 100 to the viewer terminal device 50 via the server 30.

[0115] 11(d) is a modified example of the user terminal device 10 shown in FIG. 2. The user terminal device 10 includes the out-camera 11 and the in-camera 13, but has a single image receiving element 230 that is divided into a first image receiving area 231 and a second image receiving area 232. The first image receiving area 231 receives an image from the out-camera 11, and the second image receiving area 232 receives an image from the in-camera 13. The first image receiving area 231 receives an image from the out-camera 11, and the processing unit 29 generates an image 101 that becomes a background image. The user is received in the second image receiving area 232. The processing unit 29 generates user movement information by performing image analysis on the image of the second image receiving area 232. The processing unit 29 superimposes the avatar object 102, which changes in accordance with the user's movement information detected in the second image receiving area 232, onto the image 101 generated from the image received in the first image receiving area 231, to generate composite data 100. The user terminal device 10 distributes this composite data 100 to the viewer terminal device 50 via the server 30.

[0116] The in-camera 13 may be a depth sensor 15. In this case, the second image receiving area 232 is an infrared detection unit that detects infrared rays. In this case, the number of light receiving elements is reduced to one, which makes it possible to reduce the number of parts in the user terminal device 10.

[0117] The function of overlaying the avatar object 102 on the image 101 to generate the composite data 100 may be provided by the server 30, not by the user terminal device 10. In this case, the user terminal device 10 transmits the image 101 of the subject captured by the outer camera 11 and the motion information detected by the depth sensor 15 to the server 30. The network IF 33 functions as a first acquisition unit and a second acquisition unit. The server 30 overlays the avatar object 102, which moves based on the motion information, on the image 101 to generate the composite data 100, and transmits the composite data 100 to the user terminal device 10 and distributes it to the viewer terminal device 50.

[0118] The processing performed by the server 30 may be distributed to edge servers. The composite data 100 may be generated not only by the user terminal device 10 but also by the viewer terminal device 50 or the server 30.

[0119] The user terminal device 10 does not need to have a function for performing voice recognition processing to generate text data and add a text object to the image 101. This is because it is possible to communicate with the viewer using only voice.

[0120] When the user terminal device 10 detects a specific action of the user, it is not necessary to perform a process of adding an effect object 103 related to that action in addition to the avatar object 102. This is because the intention can be communicated to the viewer even without adding the effect object 103.

[0121] The user terminal device 10 does not have to have a pedometer function. In this case, the movement of the user terminal device 10 is regarded as the movement of the user based on the detection data detected by the sensor 17, and the avatar object 102 is changed.

[0122] The sensor 17 may not be provided. Even if the avatar object 102 does not change to appear as if it is walking, as long as the user's face is at least detected and the avatar object 102, which is the user's personification, changes in accordance with the user's facial expression, the user's facial expression can be conveyed to the viewer via the avatar object 102. Also, the direction of the user's movement may be detected by changes in the images of the out-camera 11 and the in-camera 13, and the walking display of the avatar object 102 may be changed accordingly.

[0123] The sensor 17 should at least include an acceleration sensor capable of detecting acceleration. The user terminal device 10 does not need to have an outer camera 11 as long as it has an acquisition unit for acquiring video. This is because an external camera can be connected as shown in Fig. 11(a) and (b). In this case, the communication IF 21 functions as an acquisition unit such as a first acquisition unit for acquiring the image 101 and a second acquisition unit for acquiring the motion information.

[0124] The display unit 19 does not necessarily have to include a touch panel 20. This is because, if a pointing device such as a mouse is connected to the user terminal device 10 by wire or wirelessly, a position can be input using a pointer displayed on the display surface.

[0125] The user terminal device 10 may be installed with a voice changer program that processes the input voice and converts it to sound like a different voice. As an example, when distributing the composite data 100, the voice collected by an acoustic-electrical conversion device such as a microphone can be converted into a different voice by a voice changer, and the converted voice can be distributed. For example, a voice changer can change the sampling period of voice data digitized by PCM (Pulse Code Modulation) to change the pitch of the voice, delay the voice data by giving it a time delay, or synthesize the delayed voice data with the original voice data to create an echo.

[0126] The user terminal device 10 may have installed therein a voice reading program for reading text data aloud. For example, when the user terminal device 10 receives a request comment from a viewer as text data from the viewer terminal device 50, the user terminal device 10 converts the text data into voice and reads the text data aloud. The voice in this case may be, for example, a synthetic voice or the voice of a voice actor, actor, or the like. If the avatar object 102 is a character object that appears in an animation, the voice of the character's voice actor may be used. If the avatar object 102 is a person object that imitates a person such as a famous person, a famous person, or an ordinary person, the voice of the person may be used.

[0127] The composite data 100 is composed of an image 101 with an avatar object 102 superimposed thereon and sound, but it is preferable that the user's motion information, the detection data of the sensor 17, the GNSS position information, and the sound at the scene where the image 101 is captured, which are acquired at the same time as the image 101, are completely synchronized. Also, a slight time lag may be allowed as long as the image 101 and the motion information do not cause discomfort to the viewer. In such a case, the motion of the avatar object 102 and the like will be slightly shifted from the image 101. For example, as shown in FIG. 11(b), when the user terminal device 10 is separated from the first camera 206 and the second camera 207, a situation may occur in which one of the image 101 and the motion information is delayed relative to the other depending on the communication environment. In such a case, a time lag may be allowed between the image 101 and the motion information.

[0128] When live streaming is not performed, the time acquired from a GNSS, an NTP server, or the built-in clock of the user terminal device 10 or the server 30 may be used as a reference to acquire user movement information, detection data of the sensor 17, GNSS position information, and sound at the capture site of the image 101 at the same time as the reference time assigned to the image 101. As shown in Fig. 11(b) , when the user terminal device 10, the first camera 206, and the second camera 207 are separate, the same information and data may be acquired according to the built-in clock of each device to generate the composite data 100.

[0129] The user's motion information may be generated by calculating inter-frame differences or motion vectors based on the video captured by the in-camera 13. [Explanation of symbols]

[0130] 1...Distribution system, 2...Network, 3...Support member, 10...User terminal device, 11...Outer camera, 12...Outer camera IF, 13...Inner camera, 14...Inner camera IF, 15...Depth sensor, 16...Depth sensor IF, 17...Sensor, 18...Sensor IF, 19...Display unit, 20...Touch panel, 21...Communication IF, 22...Operation unit, 23...GNSS receiving unit, 24...Audio input device, 25...Audio output device, 26...Audio IF, 27...Network IF, 28...Memory unit, 28a...Distribution program, 28b...Pedometer program, 28c...Speech recognition program, 29...Processing unit, 30...Server, 31...Distribution management Database, 32...memory unit, 33...network IF, 32a...distribution management program, 34...control unit, 41...out-depth sensor, 42...out-camera IF, 50...viewer terminal device, 100...synthetic data, 101...image, 102...avatar object, 103...effect object, 104...effect object, 201...360-degree camera device, 202...first camera, 203...second camera, 206...first camera, 207...second camera, 211...semi-spherical camera, 212...lens unit, 213...captured image, 214...area, 215...area, 230...image receiving element, 231...first image receiving area, 232...second image receiving area.

Claims

[Claim 1] a first acquisition unit for acquiring image information of a first object; a second acquisition unit for acquiring motion information regarding a motion of a second subject different from the first subject; a receiving unit that receives first data transmitted by operating the external device; A data processing device comprising: a processing unit that displays on a display surface a first image based on the image information, a first virtual object that changes based on the movement information of the second subject, and a second image based on the first data received by the receiving unit.

Images