COMMUNICATION SYSTEM, COMMUNICATION DEVICE, AND COMMUNICATION METHOD

Abstract

To reduce the transmission capacity required for the entire system so as not to impede communication. [Solution] The system comprises at least a first device and a second device, wherein the first device divides the original image into multiple levels of element images of different importance based on the appearance of the subject, adjusts the information density of the element images so that the level of higher importance is relatively higher, and transmits the multiple levels of element images to the second device, and the second device receives the multiple levels of element images from the first device and synthesizes the multiple levels of element images to reconstruct a restored image.

Description

[Technical Field]

[0001] The present application relates to a communication system, a communication device, and a communication method, for example, a video communication system for conversation among multiple people. [Background technology]

[0002] In recent years, video communication systems that share video captured at multiple locations have become widespread. Furthermore, advances in filming equipment and communication technology have enabled some video communication systems to handle high-definition video, enabling high-definition video conferences. High-definition video is expected to facilitate non-verbal communication. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-155732 Summary of the Invention [Problem to be solved by the invention]

[0004] However, transmitting high-resolution video such as 4K video requires a large communication capacity (bandwidth in wireless communication). Even if high-resolution video is compressed and encoded using a video encoding method such as ITU-T 264.H, it may not be possible to transmit it as is due to limited capacity. For example, an information processing device described in Patent Document 1 acquires encoded video data captured by a camera from the camera via a network, decodes the acquired video data, and outputs the decoded, processed video data to a video processing application via a driver. A warning is displayed when the amount of stored video data exceeds a predetermined threshold, or when the difference between the time information included in the latest video data and the oldest video data among the stored video data exceeds a predetermined threshold. [Means for solving the problem]

[0005] The present application has been made to solve the above-mentioned problems, and one embodiment of a communication system is a communication system comprising at least a first device and a second device, wherein the first device comprises a first image processing unit that divides an original image into element images of multiple levels of importance based on the appearance of a subject and adjusts the information density of the element images so that the information density of the element images is relatively higher for levels of higher importance, and a first communication processing unit that transmits the element images of multiple levels to the second device, and the second device comprises a second communication processing unit that receives the element images of multiple levels from the first device, and a second image processing unit that synthesizes the element images of multiple levels to reconstruct a restored image.

[0006] In the above communication system, the first image processing unit may detect feature information indicating the morphological features and position of the subject from the original image, and the second image processing unit may reconstruct the restored image by superimposing multiple stages of the elemental images so that the morphological features are positioned at the position indicated in the feature information.

[0007] In the above communication system, the second image processing unit may convert the multiple-level element images so that the information density is equal to the highest information density among the multiple-level element images, and may reconstruct the restored image by superimposing the converted multiple-level element images.

[0008] In the above communication system, the second video processing unit may reconstruct the restored video by superimposing the converted element videos at a plurality of stages so that element videos at a stage of higher importance are given priority.

[0009] In the above communication system, the first image processing unit may detect the amount of movement of a subject of a specific type from the original image, and if the amount of movement is outside a predetermined range of amounts of movement, determine the element image showing the subject to be an element image of the lowest level, which is the lowest level of importance, and if the amount of movement is within the range of amounts of movement, determine the element image showing the subject to be a candidate for an element image of a higher level, which is the level of importance higher than the element image of the lowest level.

[0010] In the above communication system, the first image processing unit may detect the size of an image of a subject in a specific manner from the original image, and if the size is outside a predetermined size range, determine the element image showing the subject as a basic image of the lowest level, which is the lowest level of importance, and if the size is within the size range, determine the element image showing the subject as a candidate for an element image of a higher level, which is a level of importance higher than the element image of the lowest level.

[0011] In the above communication system, the subject of the particular aspect may be the upper body of a person, or a display medium that displays content.

[0012] A communication device according to a second aspect includes a video processing unit that divides an original video into a first plurality of levels of elemental video of different importance based on the appearance of the subject, and adjusts the information density of the elemental video so that the information density increases as the level of importance increases, and a communication processing unit that transmits the first plurality of levels of elemental video to another device, wherein the communication processing unit receives at least a second plurality of levels of elemental video from the other device, and the video processing unit combines the second plurality of levels of elemental video to reconstruct a restored video.

[0013] A communication method according to a third aspect is a communication system including at least a first device and a second device, in which the first device performs the steps of dividing an original image into multiple levels of element images of different importance based on the appearance of a subject, adjusting the information density of the element images so that the information density of the element images is relatively higher for levels of higher importance, and transmitting the multiple levels of element images to the second device, and the second device performs the steps of receiving the multiple levels of element images from the first device and synthesizing the multiple levels of element images to reconstruct a restored image. [Effects of the Invention]

[0014] According to the embodiment of the present application, the transmission capacity required for the entire system can be reduced so as not to impede communication. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a schematic block diagram illustrating an example of the configuration of a communication system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a schematic block diagram illustrating an example of the hardware configuration of a terminal device according to the present embodiment. [Figure 3] FIG. 2 is a schematic block diagram illustrating an example of the functional configuration of a terminal device according to the present embodiment. [Figure 4] 10 is a flowchart illustrating a video communication process according to the present embodiment. [Figure 5] FIG. 10 is a diagram illustrating a still image extracted from an original image. [Figure 6] 10A and 10B are diagrams illustrating element images extracted from a still image. [Figure 7] FIG. 10 is a diagram illustrating an example of detecting feature information from a still image. [Figure 8] FIG. 1 is a diagram illustrating an example of a basic still image that constitutes a basic video. [Figure 9] FIG. 10 is a diagram illustrating a restored still image obtained by combining an element image and a basic still image. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of combining an element video and a basic video. [Figure 11]10 is a flowchart illustrating a main part determination process according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the present application will be described with reference to the drawings. An example of the configuration of a communication system S1 according to this embodiment will be described. FIG. 1 is a schematic block diagram showing an example of the configuration of a communication system S1 according to this embodiment. The communication system S1 is configured to include a plurality of terminal devices 1. The plurality of terminal devices 1 are connected to each other via a network NW so that they can transmit and receive various data to and from each other. The network NW may be any one or a combination of the Internet, a public communication network, a local area network (LAN), a virtual personal network (VPN), a dedicated line, and the like.

[0017] In the example of Fig. 1, two terminal devices 1-1 and 1-2 are shown. The two terminal devices 1-1 and 1-2 are distinguished by assigning a sub-number -1 or -2. In the present application, the sub-number may be omitted. The number of terminal devices 1 involved in one communication may also be called the number of connections, the number of bases, etc. The number of connections is not limited to two, and may be an unspecified number of three or more, or may be fixed to a specific number.

[0018] The communication system S1 is configured as, for example, a conference system, a video call system, etc. Data transmitted and received between multiple terminal devices 1 includes video data. The transmitted and received data is used to communicate between users of the individual terminal devices 1. A dedicated communication server is connected to the network NW, and data may be transmitted between multiple terminal devices 1 via the communication server.

[0019] At least one of the multiple terminal devices 1 (for example, terminal device 1-1) acquires the video to be transmitted (referred to herein as "original video"). In this application, the terminal device 1 that transmits the video may be referred to as the "first device." The first device divides the original video into elemental videos with multiple levels of importance based on the appearance of the subject, and adjusts the information density of the elemental videos so that the level of higher importance is relatively higher. The first device transmits the adjusted elemental videos with multiple levels to another terminal device (for example, terminal device 1-2). In this application, the terminal device 1 that transmits the video may be referred to as the "second device." The second device receives the elemental images of the multiple levels from the first device, synthesizes the received elemental images of the multiple levels to reconstruct a restored image, and presents the reconstructed restored image.

[0020] Here, the first device refers to the terminal device 1 that is the source of the video, and the second device refers to the terminal device 1 that is the destination of the video, and does not refer to the hardware configuration or other functional configuration of each terminal device 1. Each terminal device 1 can either (1) perform the functions of the first device but not the functions of the second device, (2) perform the functions of the second device but not the functions of the first device, or (3) perform both the functions of the first device and the functions of the second device. In the functional configuration example of the terminal device 1 described below, it has both the functions of the first device and the functions of the second device.

[0021] Next, a configuration example of the terminal device 1 according to this embodiment will be described. Fig. 2 is a schematic block diagram showing a hardware configuration example of the terminal device 1 according to this embodiment. The terminal device 1 may be, for example, a general-purpose information terminal device such as a personal computer (PC), a tablet terminal device, or a multi-function mobile phone (including a so-called smartphone), or may be a terminal device dedicated to conferences or calls. In the following description, a case where the terminal device 1 is a PC will be exemplified.

[0022] The terminal device 1 includes a host system 10, a ROM (Read Only Memory) 22, an auxiliary storage device 23, a display 24, a camera 25, an audio system 26, a communication module 27, an input / output I / F (Interface) 28, an EC (Embedded Controller) 31, an input device 32, a power supply circuit 33, and a power switch 36.

[0023] The host system 10 is a computer system that forms the core of the terminal device 1. The host system 10 includes a processor, a main memory, and a chipset. In this application, the hardware that constitutes the host system 10 may be referred to as a "host device." The processor is a core processing device that controls the overall operation of the terminal device 1. The processor is, for example, a CPU (Central Processing Unit). The processor is a core processing device that executes arithmetic processing instructed by various instructions (commands) written in software (programs). In this application, executing processing instructed by instructions written in programs may be referred to as "executing a program" or "executing a program."

[0024] The processor may include a CPU as well as a GPU. The GPU is an arithmetic processing unit that mainly realizes functions related to image display. The GPU processes drawing commands issued by the CPU (image processing) and outputs display data indicating the obtained display information to the display 24. The GPU may be integrated with the CPU and formed on the same core, or may be formed on a core separate from the CPU.

[0025] The main memory is a writable memory used as a read area for the processor's execution program and as a work area for writing the processing data of the execution program. The main memory is composed of, for example, multiple DRAM (Dynamic Random Access Memory) chips. The processor and main memory are the minimum hardware that constitutes the host system 10.

[0026] The chipset 21 includes multiple controllers and can be connected to multiple devices to input and output various types of data. The controller included in the chipset 21 may be, for example, a Universal Serial Bus (USB), a Serial Peripheral Interface (SPI) bus, a PCI-Express bus, or the like.

[0027] The ROM 22 mainly stores firmware. The firmware stored in the ROM 22 includes a BIOS (Basic Input-Output System) and other firmware related to individual devices. The ROM 22 includes a rewritable nonvolatile memory such as an EEPROM (Electrically Erasable Programmable Read Only Memory) or a flash ROM.

[0028] The auxiliary storage device 23 stores various data used in the processing of the host system 10, various data acquired by the processing, various programs, etc. The auxiliary storage device 23 may be, for example, an SSD (Solid State Drive), an HDD (Hard-disk Drive), etc.

[0029] The display 24 displays a display screen based on display data input from the host system 10. The display 24 may be, for example, a liquid crystal display (LCD) or an organic light emitting diode (OLED) display.

[0030] Camera 25, under the control of host system 10, captures images of subjects appearing within its field of view. Camera 25 is a video camera capable of capturing video, i.e., moving images. Video generally consists of still images captured at regular intervals in chronological order. Camera 25 outputs video data representing the captured video to host system 10.

[0031] The audio system 26 is equipped with an audio codec and performs input and output of audio data. Under the control of the host system 10, the audio codec converts analog input audio signals input from a microphone into digital input audio data and outputs the input audio data obtained through the conversion to the host system 10. The microphone detects audio propagating therethrough and outputs an input audio signal indicating the detected audio to the audio system 26. The audio codec is equipped with a decoder that converts digital output audio data output from the host system 10 into an analog output audio signal and outputs the output audio signal obtained through the conversion to a speaker. The speaker presents audio based on the output audio signal input from the audio system 26. The microphone and speaker may be provided in the terminal device 1, or one or both of them may be detachably connected to the terminal device 1, or may be separate from the terminal device 1.

[0032] The communication module 27 connects to a communication network so as to be able to transmit and receive various types of data wirelessly or via a wire. The communication module 27 communicates various types of data with other devices connected to the communication network. The communication module 27 is, for example, a wireless LAN module that connects to a wireless LAN.

[0033] The input / output I / F 28 connects to various devices via wire or wirelessly to input and output data, and includes, for example, a USB connector for inputting and outputting data via wire in accordance with the USB standard.

[0034] The EC 31 is a controller that monitors and controls the operations of various devices connected thereto, regardless of the operating state of the host system 10. The EC 31 includes a CPU, ROM, RAM, a timer, and an input / output I / F, separate from the host system 10. Devices with data transfer speeds slower than those of devices connected to the host system 10 can be connected to the EC 31. In the example of FIG. 2, an input device 32, a power supply circuit 33, and a power switch 36 are connected to the EC 31.

[0035] The EC 31 reads predetermined firmware from its own ROM and executes the read firmware to provide its functions. Instead of its own ROM, firmware may be stored in the ROM 22 in advance and the read firmware may be executed. However, it is necessary to start up the ROM 22 when the EC 31 is started up.

[0036] The input device 32 detects a user operation, generates an operation signal in accordance with the detected operation, and outputs the operation signal to the EC 31. The input device 32 may be, for example, a keyboard, a touchpad, or the like.

[0037] The power supply circuit 33 includes a voltage converter and a charger. The voltage converter converts the voltage of DC power supplied from an external power supply or a battery (not shown) into a voltage required for the operation of each device constituting the terminal device 1, and supplies power having the converted voltage to the destination device. The power supply circuit 33 supplies power to the device under the control of the EC 31. The charger charges the battery with the remaining power that is supplied from the external power source and not consumed by each device. When power is not supplied from the external power source or when the power supplied from the external power source does not meet the demand, the charger supplies the power discharged from the battery to each device. The battery charges with power supplied from the power supply circuit 33 or discharges power stored in itself to the power supply circuit 33. The battery may be, for example, a lithium-ion battery, a sodium-ion battery, or the like.

[0038] Each time the power switch 36 is pressed, it controls the state of power supply to the host system 10 to either power on (Power ON) or power off (Power OFF). When a press operation is received, the power switch 36 outputs a press signal indicating the press to the EC 31. When the terminal device 1 is powered off and a press signal is input from the power switch 36, the EC 31 causes the power circuit 33 to start supplying power to each device of the terminal device 1 (power on). When power is supplied to the terminal device 1 and a press signal is input from the power switch 36, the EC 31 causes the host system 10 to execute a stop process (shutdown).

[0039] Next, an example of the functional configuration of the terminal device 1 will be described. The functions of the terminal device 1 are realized by the processor of the host system 10 executing various programs in cooperation with the main memory and other hardware. FIG. 3 is a schematic block diagram showing an example of the functional configuration of the terminal device 1 according to this embodiment. The terminal device 1 includes a video processing unit 12, a communication processing unit 14, and an output processing unit 16. The video processing unit 12 includes an analysis unit 122, a multiplexing unit 124, and a synthesis unit 126.

[0040] The analysis unit 122 receives video data from the camera 25 (FIG. 2). The analysis unit 122 analyzes the original video shown in the video data to detect a display area of ​​a subject with a specific appearance. The appearance of the subject is determined as its type or a combination of the type and an action state. The analysis unit 122 performs known image recognition processing on the original video, which is composed of, for example, multiple frames of still images, to determine the type of subject appearing in the original video and the change in its display area over time. In the image recognition processing, for example, a machine learning model such as LSTM (Long Short-Term Memory) or RNN (Recurrent Neural Network) is used. The analysis unit 122 classifies the determined appearance of the subject into multiple predetermined stages of elemental video. The analysis unit 122 has stages corresponding to each appearance of the subject set in advance, and classifies the appearance of each individual subject into one of the multiple stages. Each stage has a different importance in communication between users.

[0041] For example, a person's face, whether moving or not, is more important than other parts of the person or other subjects. Certain organs of a person's face, particularly the eyes and lips, are more important than other organs. Even among other organs, multiple parts of a moving head are more important than those in a stationary state. Changes in the relative positions of these multiple parts represent the person's facial expression. Furthermore, the head, hands, or arms of a moving person are more important than those in a stationary state. The movements of the head, hands, or arms are formed as gestures. In other words, the posture of a person's upper body provides clues to non-verbal communication.

[0042] Furthermore, display media that allow content to be displayed on a display surface through user operation are more important than other subjects. This is because such display media can be used as a means to complement information transmission in video communication between users. The displayed content consists of any one or a combination of characters, symbols, and figures. Examples of such display media include touch panels and bulletin boards. A touch panel may be a standalone device or may be part of an information terminal device such as a smartphone that primarily has other functions. A bulletin board may be in the form of a blackboard, whiteboard, notebook, or the like, and the display mechanism may be electronic or non-electronic. Such a display medium may have, for example, a display surface that displays the trajectory of a writing action (drawing input).

[0043] However, a display medium that does not display content due to an operation is considered less important. More specifically, a display medium that is being held by a person's hand or whose display surface is being pointed at may be determined to be higher than a display medium that is not being held or whose display surface is not being pointed at. Note that background objects such as walls and windows, equipment other than the display medium, clothing, etc. may be classified as the lowest level of importance because these objects do not contribute to communication between users. When there are multiple subjects in the original video that are classified as the lowest level of importance, the analysis unit 122 may not distinguish between these multiple subjects and may instead detect the collection of these multiple subjects as the background.

[0044] The analysis unit 122 analyzes the original image shown in the video data to detect feature information indicating predetermined morphological features appearing in the subject. The analysis unit 122 may explicitly include information on the position of the detected morphological features in the feature information. The analysis unit 122 detects feature points and / or edges as feature information using a known image processing technique. The detection of feature information may be performed as part of image recognition processing using a machine learning model, or may be performed independently. The analysis unit 122 may identify element images so that one or more feature points are included for one subject. The analysis unit 122 may identify element images so that part or all of an edge is included in the outer edge of one subject.

[0045] Feature points are detected as minute parts that differ in color or shade from their surroundings. Feature points are sometimes called landmarks. A feature point corresponds to a group of spatially adjacent pixels, each with a signal value gradient steeper than a predetermined reference value, with a diameter of a predetermined size or less. The signal values ​​to be processed may be either luminance values ​​or color signal values. Typical feature points include dimples, scars, moles, corners of the eyes, corners of the mouth, and cleft lips on the face.

[0046] Edges are detected as contours or lines with large changes in color or shade. An edge is a region where a predetermined number of adjacent pixels, each with a signal value gradient steeper than a predetermined reference value, are spatially adjacent, and corresponds to a region whose length in a specific direction is longer than a predetermined length and whose width in a direction intersecting the length direction is sufficiently thinner than the length. Typical edges include the jaw, which forms the base of the head, the side edges, the top of the head or the outer edge of the hair, and the outer edges of the hands or arms.

[0047] The analysis unit 122 adjusts the information density of each of the multiple levels of elemental video using known image processing techniques so that the more important the level, the higher the information density. The information density of the video is determined by the resolution, bit depth, and frame rate. Generally, the number of pixels per frame is used as an indicator of resolution. The greater the number of pixels per frame, the higher the information density. Bit depth is the number of bits that express the signal value of each pixel. The higher the bit depth, the higher the information density. The frame rate corresponds to the number of frames per second. The higher the frame rate, the shorter the frame interval (period) and the higher the information density. However, the analysis unit 122 reduces the total information amount of the multiple levels of elemental video less than the information amount of the original image. Therefore, the lower the level of importance, the greater the rate of reduction in information density. The analysis unit 122 outputs the elemental images and feature information at multiple stages to the multiplexing unit 124 .

[0048] The multiplexing unit 124 multiplexes the individual elemental images and feature information input from the analysis unit 122 . The multiplexing unit 124 performs encoding processing on each of the elemental videos at the multiple levels using a predetermined video encoding method to generate encoded data. The predetermined video encoding method may be, for example, an encoding method such as the method specified by ITU-T H.264 (AVC: Advanced Video Coding) or the method specified by ITU-T H.265 (VVC: Versatile Video Coding). The multiplexing unit 124 multiplexes the encoded data at the multiple levels, including the feature information, by associating them with each other to generate multiplexed data. The multiplexing unit 124 outputs the generated multiplexed data to the communication processing unit 14. The output multiplexed data is transmitted to the destination terminal device 1 (sometimes referred to as the "destination device" in this application) using the communication processing unit 14.

[0049] The combining unit 126 receives multiplexed data from the destination device via the communication processing unit 14. The combining unit 126 separates the encoded data and feature information for each of the multiple stages from the input multiplexed data. The combining unit 126 performs a decoding process on the separated encoded data for each stage using a predetermined video decoding method to restore the elemental videos. The predetermined video decoding method may be any method that corresponds to the video encoding method used for conversion to encoded data.

[0050] The synthesis unit 126 identifies the position so that the morphological features of the subject are positioned corresponding to the positions indicated in the feature information, and then superimposes the elemental images of multiple levels to reconstruct a restored image. Before superimposing the elemental images of multiple levels, the synthesis unit 126 converts the information densities of the elemental images of multiple levels using known image processing techniques to achieve a single common set of data densities, and acquires the converted elemental images as converted elemental images. The synthesis unit 126 spatially and temporally interpolates the signal values ​​of each pixel of the elemental images of each frame constituting the first-level elemental image so that the resolution, bit depth, and frame rate per unit area of ​​a certain first-level elemental image are equal to the resolution, bit depth, and frame rate of a more important second-level elemental image. For example, the highest information density among the multiple levels may be applied as the single common set of data densities. Uniformly aligning the data densities between levels facilitates manipulation of pixel values ​​for each frame and pixel. This is also advantageous for identifying the positions of the elemental images based on morphological features.

[0051] A region of a certain first-stage conversion element image whose position has been specified may overlap with a portion of a region of a second-stage conversion element image, which is another stage. Therefore, when superimposing multiple stages of conversion element images whose positions have been specified, the composition unit 126 prioritizes the conversion element image of a higher importance stage. That is, when signal values ​​are set for each of two or more stages of conversion element images for a given pixel, the composition unit 126 adopts the signal value associated with the conversion element image of the highest importance stage and discards the signal values ​​associated with the other stages of conversion element images. In this case, the composition unit 126 sequentially overwrites the conversion element image of the lowest stage, which is the least important stage, with the conversion element image of the higher importance stage.

[0052] The video encoding and decoding methods exemplified above can encode and decode video having rectangular regions, but cannot process video having arbitrary shapes. Therefore, the processing target is video having a rectangular region inscribed around the subject. As a result, a gap between the rectangular region of the superimposed element video and the subject in the restored video may be visible, causing a sense of discomfort to the user.

[0053] Therefore, the composition unit 126 may perform image recognition processing on a rectangular area including element images with a higher importance level than the lowest level to extract element videos representing the subject and eliminate gaps between the rectangular area and the extracted element videos. The composition unit 126 uses the above-described method to superimpose the extracted element videos to compose a restored image. The gaps between the rectangular area and the element videos will no longer appear in the restored image. The synthesis unit 126 outputs video data representing the synthesized restored video to the output processing unit 16.

[0054] The communication processing unit 14 is connected to the network NW and identifies the terminal device 1 as the destination device. The destination device may be identified using any identification information such as a URL (Universal Resource Locator) or a SIP-URI (Session Initiation Protocol-Universal Resource Identifier). For example, when a connection request is input from a destination device to the device itself, the communication processing unit 14 displays on the display 24 a query screen for querying whether or not to permit connection with the destination device. When an operation signal input from the input device 32 indicates an "allow" button arranged on the query screen, the communication processing unit 14 determines that the connection is permitted. The communication processing unit 14 then transmits an acknowledgement to the destination device that sent the connection request. At this time, a connection is established between the device itself and the destination device.

[0055] The communication processing unit 14 determines that the connection has been refused when the reject button on the inquiry screen is pressed by an operation signal input from the input device 32, or when the accept button is not pressed even after a predetermined waiting time has elapsed since the inquiry screen was displayed. The communication processing unit 14 then transmits a reject response to the destination device that sent the connection request. At this time, the connection between the device and the destination device is not established.

[0056] The output processing unit 16 may identify the destination device and display on the display 24 an operation screen for instructing the start of communication. At this time, the communication processing unit 14 may receive an operation signal indicating a connection request to the destination device from the input device 32. When the operation signal indicating a connection request is input from the input device 32, the communication processing unit 14 transmits the connection request to the destination device indicated by the operation signal. When the communication processing unit 14 receives an acknowledgment response to the connection request from the destination device, the communication processing unit 14 determines that the connection with the destination device is permitted. At this time, a connection is established between the device and the destination device. When the communication processing unit 14 receives a rejection response to the connection request from the destination device, or when the communication processing unit 14 does not receive an acknowledgment response even after a predetermined waiting time has elapsed since transmitting the connection request, the communication processing unit 14 determines that the connection with the destination device is not permitted. At this time, the connection between the device and the destination device is not established.

[0057] The communication processing unit 14 transmits and receives communication data to and from the destination device when a connection with the destination device is established. The communication processing unit 14 transmits the multiplexed data input from the video processing unit 12 to the destination device. The communication processing unit 14 may include in the multiplexed data one or both of the audio data input from the audio system 26 and application screen data showing a screen (sometimes referred to herein as an "application screen") obtained by executing another application program (sometimes referred to herein as an "application") and transmit the multiplexed data to the destination device. The application may provide, for example, a chat function, a document display function, or the like.

[0058] When receiving multiplexed data from a destination device, the communication processing unit 14 outputs the received multiplexed data to the combining unit 126. If audio data is multiplexed in the received multiplexed data, the communication processing unit 14 separates the audio data from the multiplexed data and outputs the separated audio data to the audio system 26 via the output processing unit 16. If application screen data is multiplexed in the received multiplexed data, the communication processing unit 14 separates the application screen data from the multiplexed data and outputs the separated application screen data to the output processing unit 16.

[0059] The output processing unit 16 executes processing for outputting various information related to communication with the destination device. The output processing unit 16 constructs a display screen including the restored video shown in the video data input from the synthesis unit 126. The output processing unit 16 outputs display data showing the constructed display screen to the display 24. The display 24 displays the display screen including the restored video based on the display data.

[0060] When application screen data is input from the communication processing unit 14, the output processing unit 16 may construct a display screen including the application screen shown in the application screen data. When audio data is input from the communication processing unit 14, the output processing unit 16 may output the input audio data to the audio system 26, causing the audio to be presented from a speaker. The output processing unit 16 may output display data showing various setting screens to the display 24.

[0061] Next, the video communication processing according to this embodiment will be described. FIG. 4 is a flowchart illustrating the video communication processing according to this embodiment. The video communication processing illustrated in FIG. 4 shows a series of processes from capturing an original video in terminal device 1-1 to displaying a restored video in terminal device 1-2. The following description mainly focuses on the case where the number of levels of elemental video is two. The level with the lowest importance is sometimes referred to as the "lowest level," a subject or its image relating to the lowest level as a "non-essential part," and video showing a non-essential part as a "non-essential part video." A level with higher importance than the lowest level is sometimes referred to as a "high level," a subject or its image relating to the high level as a "main part," and video showing a main part as a "main part video." The number of essential part images detected from the original image is not necessarily limited to one, but may be two or more, or may be zero.

[0062] The terminal device 1-1 executes the processes of steps S102 to S108. (Step S102) The camera 25 captures an image. (Step S104) The analysis unit 122 uses the video captured by the camera 25 as the original video and detects elemental videos each having a specific type of subject as a key part from the original video using a predetermined model. The analysis unit 122 also detects feature points from the original video that indicate predetermined morphological characteristics of the subject. (Step S106) The analysis unit 122 detects elemental images representing non-essential parts from the original video using a model separate from that used for detecting essential parts. (Step S108) The analysis unit 122 increases the information density of the essential part video showing the essential parts compared to the non-essential part video showing the non-essential parts, and the communication processing unit 14 transmits multiplexed data, which is obtained by multiplexing the encoded data of the essential part video and the non-essential part video and feature information showing feature points, to the terminal device 1-2 via the network NW.

[0063] The terminal device 1-2 executes the processes of steps S110 to S116. (Step S110) The communication processing unit 14 receives the multiplexed data from the terminal device 1-1. The combining unit 126 separates the coded data of the main video and the non-main video, and the feature information indicating the feature points, from the multiplexed data. (Step S112) The synthesis unit 126 adjusts the information density of the non-essential part video so that it becomes equal to the information density of the essential part video. (Step S114) The synthesis unit 126 reconstructs the restored video by superimposing the main part video so that the positions of the feature points of the main part video are positioned at the positions of the feature points indicated in the feature information in the adjusted non-main part video. (Step S116) The output processing unit 16 displays the reconstructed restored video on the display 24. Thereafter, the processing of FIG. 4 ends.

[0064] In the communication system S1, the terminal device 1-2 may also transmit video of the terminal device 1-1. In this case, the terminal device 1-2 further performs the processes of steps S102 to S108, and the terminal device 1-1 performs the processes of steps S110 to S116.

[0065] Next, an example of video or image processing will be shown. Fig. 5 shows an example of a still image that constitutes the original video. The still image shown in Fig. 5 is captured by the camera 25 of the terminal device 1-1 during communication with the terminal device 1-2. This still image shows the front of the head and chest of the user of the terminal device 1-1 at a certain time. FIG. 6 illustrates still images constituting the essential part video. The illustrated still images show the left eye, right eye, and lips of the user's face as essential parts detected by the analysis unit 122 from the still image of FIG. 5. The left eye, right eye, and lips are organs whose shape and position change significantly depending on the user's emotions. These organs tend to attract the attention of the other party during communication. Unlike non-essential part video, the essential part video prevents or alleviates a decrease in information density, so communication is not impaired.

[0066] Figure 7 shows an example of feature information detected from an original image. The illustrated feature information includes feature points and edges detected from the still image illustrated in Figure 5. The feature points are indicated by crosses at the corners of the eyes, the corners of the lips, and the cleft lip. The edges are indicated by curves at the bottom of the face, including the chin. 8 illustrates still images constituting the non-essential part video. The illustrated still images are detected from the original video by the analysis unit 122 and have lower resolution than the essential part video. The illustrated still images show a background consisting mainly of windows. Because the background is not noticed by the user of the remote device, the non-essential part video showing the background is allowed to have lower information density than the essential part video.

[0067] 9 illustrates still images constituting the restored video. The illustrated restored video is obtained by superimposing the essential part video on the non-essential part video in the synthesis unit 126. The essential part video is superimposed on the non-essential part video so that the feature points of the essential part video are positioned at the positions indicated by the feature information.

[0068] Next, a specific example of combining an essential part video and a non-essential part video will be described. FIG. 10 is an explanatory diagram illustrating the combining of an essential part video and a non-essential part video. Here, the frame rate of the essential part video is three times that of the non-essential part video. The combining unit 126 interpolates still images for each frame that constitute the non-essential part video so that the frame rate becomes three times faster. When interpolating still images, the combining unit 126 may repeat the still images that constitute the non-essential part video three times at the same frame period as the elemental video, or may generate still images by performing morphing using still images for each frame. FIGS. 10(a) to 10(d) show the non-essential part video and the essential part video for each frame from time t=t0 to t0+3ΔT (=t0+ΔT; ΔT and ΔT indicate the frame intervals between the essential part video and the non-essential part video, respectively) using thin and thick lines. The edges that form the outline of the essential part video are used as the morphological characteristics of the essential part video. The still image forming the main part video is superimposed on the still image forming the non-element video, whose outline is positioned at the position indicated in the feature information. Figures 10(a) to 10(d) show that the subject appearing in the main part video moves sequentially to the right from time t=t0 to t0+3ΔT. Therefore, even if the frame rates of the main part video and the non-element part video are different, the movement of the subject shown in the main part video is reproduced smoothly on the non-element part video with the adjusted frame rate.

[0069] Note that a key part determined uniformly based on the type of subject does not necessarily contribute to the communication of information between users. Therefore, the analysis unit 122 may determine a subject whose movement amount between frames is equal to or greater than a predetermined reference value of the movement amount as a non-key part. This is because a subject moving at high speed is unlikely to notice a decrease in image quality even if the information density is reduced, and the subject does not contribute to the communication of precise information. Furthermore, the analysis unit 122 may exclude subjects whose size is outside a predetermined range from the key part, and determine subjects whose size is within the predetermined range as candidates for the key part.

[0070] Next, an example of the main part determination process will be described with reference to Fig. 11. Fig. 11 is a flowchart illustrating the main part determination process according to this embodiment. (Step S202) Analysis unit 122 identifies a subject appearing in the original video acquired from camera 25. If the identified subject is a predetermined type of subject (step S202 YES), analysis unit 122 proceeds to the processing of step S204. If the identified subject is a type of subject different from the predetermined type (step S202 NO), the display area of ​​the subject is determined to be a non-essential part, and the processing of FIG. 11 ends.

[0071] (Step S204) If the amount of movement of a predetermined type of subject from the subject appearing in the immediately preceding frame is equal to or greater than a predetermined reference amount (step S204 YES), the analysis unit 122 determines that the display area of ​​the subject is a non-essential part, and ends the processing of Fig. 11. If the amount of movement is less than the predetermined reference amount (step S204 NO), the processing proceeds to step S206.

[0072] (Step S206) The analysis unit 122 determines whether the size of the predetermined type of subject is within a predetermined size range (for example, 1 / 32 to 1 / 64 of one side of the display screen). If the size is within the predetermined range (step S206 YES), the analysis unit 122 determines that the image of the subject is an essential part. Thereafter, the processing of FIG. 11 ends. If the size is smaller or larger than the predetermined range (step S206 NO), the analysis unit 122 determines that the display area of ​​the subject is a non-essential part, and the processing of FIG. 11 ends.

[0073] As explained above, encoding and transmitting high-resolution video as is requires a large transmission capacity (i.e., bandwidth). When encoding and transmitting 4K video (3840 pixels x 2160 pixels) at 30 frames per second based on the ITU-T H.264 standard, the required communication capacity is 16 Mbps. However, in this embodiment, the 4K video is divided into essential video and non-essential video as the original video, and the information density of the essential video is maintained and the information density of the non-essential video is reduced from the original video. This reduces the communication capacity of the entire video without significantly reducing subjective quality. For example, when the essential video, which forms part of the area of ​​one frame, is encoded and transmitted at 540 pixels x 960 pixels and 30 frames per second, and the non-essential video is encoded and transmitted at 240 pixels x 135 pixels and 10 frames per second, the required transmission capacity is 1 Mbps for the essential video and 0.2 Mbps for the non-essential video. Overall, the transmission speed is 1.2Mbps, which is less than 1 / 12 of the transmission capacity required when transmitting 4K video directly.

[0074] The above description mainly illustrates an example in which the number of connections in the communication system S1 is two and the system is applied to communication involving the transmission of video data between two terminal devices 1. However, this is not limiting. The communication system S1 may also be applied to communication between three or more terminal devices 1 with three or more connections. In this case, at least one of the three or more terminal devices 1 is designated as a first device, and the remaining terminal devices are designated as second devices. The first device transmits common multiplexed data based on video acquired by the first device to each of the second devices. The first device receives multiplexed data based on video acquired by each of the second devices from each of the second devices. The first device is capable of synthesizing restored video based on each multiplexed data. The first device may present all of the synthesized restored video in parallel, or may present only a portion of the restored video. The communication processing unit 14 can use the above method to identify three or more terminal devices 1 participating in one communication.

[0075] The analysis unit 122 may reduce the amount of information of multiplexed data per transmission source device as the number of connections participating in one communication increases. Here, the analysis unit 122 may reduce the information density of the non-essential part video or the information density of the essential part video as the number of connections increases. Multiple levels of subject types that are candidates for essential parts may be set in advance in the analysis unit 122, and the range of subjects that are candidates for essential parts may be narrowed as the number of connections increases. When the number of connections exceeds a predetermined upper limit of the number of connections, the analysis unit 122 may not perform essential part analysis, but may convert the entire original video into non-essential part video and stop transmitting the essential part video.

[0076] The communication processing unit 14 may determine the presence or absence of speech using known audio processing technology based on the audio data input to the unit. The communication processing unit 14 may transmit multiplexed data based on video acquired during a speech period determined to be during speech, and may stop transmitting multiplexed data based on video acquired during a non-speech period when speech is not determined to be occurring. Furthermore, the communication processing unit 14 may stop transmitting the essential video during a speech period, treating the entire original video as non-essential video.

[0077] As described above, the communication system S1 according to this embodiment includes at least a first device (e.g., terminal device 1-1) and a second device (e.g., terminal device 1-2). The first device includes a first video processing unit (e.g., analysis unit 122 of video processing unit 12) that divides an original video into element videos of different levels of importance based on the appearance of a subject and adjusts the information density of the element videos so that the information density of the element videos is relatively higher for higher levels of importance, and a first communication processing unit (e.g., communication processing unit 14) that transmits the element videos of the different levels to the second device. The second device includes a second communication processing unit (e.g., communication processing unit 14) that receives the element videos of the different levels from the first device, and a second video processing unit (e.g., synthesis unit 126 of video processing unit 12) that synthesizes the element videos of the different levels to reconstruct a restored video and outputs the restored video. The specific type of subject may be the upper body of a person (for example, a head, a hand, an arm, etc.) or a display medium that displays content (for example, a touch panel, a bulletin board, etc.). According to this configuration, elemental images with reduced information density according to importance are transmitted from the first device to the second device, and a restored image is output that is reconstructed by compositing elemental images with elemental images over multiple stages. This makes it possible to ensure quality by relatively increasing the information density for subjects that are important for communication while reducing the communication capacity required for the entire communication system S1, thereby maintaining smooth communication through video communication between the first device and the second device.

[0078] The first image processing unit may detect feature information indicating predetermined morphological features (e.g., feature points, edges) and positions of the subject from the original image, and the second image processing unit may reconstruct the restored image by superimposing multiple stages of elemental images so that the morphology is positioned at the position indicated in the feature information. With this configuration, the position where an elemental video that is part of the video to be transmitted should be superimposed is identified based on the morphological features indicated by the feature information. Also, by omitting the process of analyzing the feature information in the second video processing unit, processing efficiency can be improved.

[0079] The second image processing unit may convert the element images with multiple levels so that the information density is equal to the highest information density among the element images with multiple levels, and may reconstruct the restored image by superimposing the converted element images with multiple levels. With this configuration, the information density is standardized among the elemental images at multiple levels so that it is equal to the highest information density among the elemental images at multiple levels, thereby reducing the load associated with the processing involved in superimposing the elemental images at multiple levels and ensuring the quality of the restored image.

[0080] The second video processing unit may reconstruct the restored video by superimposing element videos at a plurality of stages so that element videos at a stage with higher importance are given priority. With this configuration, even if elemental videos overlap between multiple stages, elemental videos of more important stages are reflected in the restored video, and quality degradation due to information loss caused by the overlapping of elemental videos can be reduced.

[0081] The first image processing unit may detect the amount of movement of a subject of a specific type from the original image, and if the amount of movement is outside a predetermined range of movement amounts, determine the element image showing the subject to be an element image of the lowest level, which is the lowest level of importance, and if the amount of movement is within the predetermined range of movement amounts, determine the element image showing the subject to be a candidate for an element image of a higher level, which is the level of importance higher than the element image of the lowest level. According to this configuration, among subjects of a specific aspect, images of subjects whose movement amount is within a predetermined range are candidates for elemental images with a high level, and images of subjects whose movement amount exceeds the predetermined range are candidates for elemental images with a lowest level. By assigning the lowest level of information density to images of subjects whose movement amount exceeds the predetermined range and does not contribute to communication, it is possible to reduce transmission capacity without impairing communication.

[0082] The first image processing unit detects the size of an image of a subject in a specific state from the original image, and if the size is outside a predetermined size range, determines the element image showing the subject to be a basic image of the lowest level, which is the lowest level of importance, and if the size is within a predetermined size range, determines the element image showing the subject to be a candidate for an element image of a higher level, which is a level of importance higher than the element image of the lowest level. With this configuration, subject images within a specific range of size are candidates for elemental images with a higher level, and subject images with sizes exceeding a predetermined range are candidates for elemental images with the lowest level. By assigning the lowest level of information density to subject images with sizes exceeding a predetermined range that do not contribute to communication, it is possible to reduce transmission capacity without impairing communication.

[0083] Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configurations are not limited to the above-described embodiments, and the present invention also includes designs that do not deviate from the gist of the present invention. The configurations described in the above-described embodiments can be combined in any manner. [Explanation of symbols]

[0084] S1...communication system, 1 (1-1, 1-2)...terminal device, 10...host system, 12...video processing unit, 14...communication processing unit, 16...output processing unit, 22...ROM, 23...auxiliary storage device, 24...display, 25...camera, 26...audio system, 27...communication module, 28...input / output I / F, 31...EC, 32...input device, 33...power supply circuit, 36...power switch, 122...analysis unit, 124...multiplexing unit, 126...combining unit, NW...network

Claims

1. A communication system comprising at least a first device and a second device, The first device is Dividing the original video into elemental videos with different levels of importance based on the state of the subject; a first image processing unit that adjusts the information density of the elemental images so that the information density of the elemental images becomes relatively higher as the level of importance increases; a first communication processing unit that transmits the elemental images of the plurality of stages to the second device; The second device is a second communication processing unit that receives the elemental images of the plurality of stages from the first device; a second image processing unit that reconstructs a restored image by synthesizing the element images at a plurality of stages, The first image processing unit detecting feature information indicating the shape and position of the subject from the original image; Detecting the amount of movement of a subject in a specific manner from the original video; If the amount of movement is outside a predetermined range of amounts of movement, the element image showing the subject is determined to be an element image of the lowest level, which is the lowest level of importance; If the amount of movement is within the range of the amount of movement, the element image showing the subject is determined to be a candidate for a high-level element image that is more important than the element image of the lowest level; The second image processing unit reconstructing the restored image so that the morphological features are positioned at positions indicated by the feature information; When reconstructing the restored image, the element images of the plurality of levels are converted so that the information density is equal to the highest information density among the element images of the plurality of levels, and the converted element images of the plurality of levels are superimposed. Communication system.

2. A communication system comprising at least a first device and a second device, The first device is Dividing the original video into elemental videos with different levels of importance based on the state of the subject; a first image processing unit that adjusts the information density of the elemental images so that the information density of the elemental images becomes relatively higher as the level of importance increases; a first communication processing unit that transmits the elemental images of the plurality of stages to the second device; The second device is a second communication processing unit that receives the elemental images of the plurality of stages from the first device; a second image processing unit that reconstructs a restored image by synthesizing the element images at a plurality of stages, The first image processing unit detecting feature information indicating the shape and position of the subject from the original image; Detecting the size of an image of a subject in a specific aspect from the original image; If the size is outside a predetermined range, the elemental image showing the subject is determined to be a basic image of the lowest level, which is the level of least importance; If the size is within the size range, the element image showing the subject is determined to be a candidate for a high-level element image that is a level of importance higher than the element image of the lowest level; The second image processing unit reconstructing the restored image so that the morphological features are positioned at positions indicated by the feature information; When reconstructing the restored image, the element images of the plurality of levels are converted so that the information density is equal to the highest information density among the element images of the plurality of levels, and the converted element images of the plurality of levels are superimposed. Communication system.

3. The second image processing unit reconstructs the restored image by superimposing the element images at a plurality of stages so that an element image at a stage having a higher importance is prioritized.

3. The communication system according to claim 1 or 2.

4. The subject of the specific aspect is the upper body of a person or a display medium that displays content.

3. The communication system according to claim 1 or 2.

5. a video processing unit that divides the original video into a first plurality of levels of element video having different importance based on the state of a subject, and adjusts the information density of the element video so that the information density of the element video increases as the level of importance increases; a communication processing unit that transmits the first elemental image of multiple levels to another device, The video processing unit includes: detecting feature information indicating the shape and position of the subject from the original image; Detecting the amount of movement of a subject in a specific manner from the original video; If the amount of movement is outside a predetermined range of amounts of movement, the element image showing the subject is determined to be an element image of the lowest level, which is the lowest level of importance, among the first plurality of levels; If the amount of movement is within the range of the amount of movement, the element image showing the subject is determined to be a candidate for an element image of a higher level among the first plurality of levels, the element image being of a higher level of importance than the element image of the lowest level; The communication processing unit receiving at least a second plurality of stages of elemental images from the other device; The video processing unit includes: reconstructing a restored image by synthesizing the second plurality of stages of element images; reconstructing the restored image so that the morphological features are arranged at positions indicated by feature information indicating the morphological features and positions of the subject appearing in the second plurality of stages of elemental images; When reconstructing the restored image, the element images of the plurality of levels are converted so that the information density is equal to the highest information density among the element images of the plurality of levels, and the converted element images of the plurality of levels are superimposed. Communication equipment.

6. a video processing unit that divides the original video into a first plurality of levels of element video having different importance based on the state of a subject, and adjusts the information density of the element video so that the information density of the element video increases as the level of importance increases; a communication processing unit that transmits the first elemental image of multiple levels to another device, The video processing unit includes: detecting feature information indicating the shape and position of the subject from the original image; Detecting the size of an image of a subject in a specific aspect from the original image; If the size is outside a predetermined range, the elemental image showing the subject is determined to be a basic image of the lowest level, which is the level of least importance, among the first plurality of levels; If the size is within the size range, the element image showing the subject is determined to be a candidate for an element image of a higher level among the first plurality of levels, the level being higher in importance than the element image of the lowest level; It is determined as a candidate for a high-level element image, which has a higher level of importance. The communication processing unit receiving at least a second plurality of stages of elemental images from the other device; The video processing unit includes: reconstructing a restored image by synthesizing the second plurality of stages of element images; reconstructing the restored image so that the morphological features are arranged at positions indicated by feature information indicating the morphological features and positions of the subject appearing in the second plurality of stages of elemental images; When reconstructing the restored image, the element images of the plurality of levels are converted so that the information density is equal to the highest information density among the element images of the plurality of levels, and the converted element images of the plurality of levels are superimposed. Communication equipment.

7. In a communication system comprising at least a first device and a second device, The first device is A step of dividing the original video into a plurality of element videos having different levels of importance based on the aspect of the subject; adjusting the information density of the elemental images so that the information density is relatively higher for stages with higher importance; transmitting the elemental images of the plurality of stages to the second device; The second device is receiving the elemental images of the plurality of stages from the first device; and reconstructing a restored image by synthesizing the element images at a plurality of stages.

1. A communication method comprising: The first device is detecting feature information indicating the shape and position of the subject from the original image; Detecting the amount of movement of a subject in a specific manner from the original video; If the amount of movement is outside a predetermined range of amounts of movement, the element image showing the subject is determined to be an element image of the lowest level, which is the lowest level of importance; If the amount of movement is within the range of the amount of movement, the element image showing the subject is determined to be a candidate for a high-level element image that is more important than the element image of the lowest level; The second device is reconstructing the restored image so that the morphological features are positioned at positions indicated by the feature information; When reconstructing the restored image, the element images of the plurality of levels are converted so that the information density is equal to the highest information density among the element images of the plurality of levels, and the converted element images of the plurality of levels are superimposed. Communication method.

8. In a communication system comprising at least a first device and a second device, The first device is A step of dividing the original video into a plurality of element videos having different levels of importance based on the aspect of the subject; adjusting the information density of the elemental images so that the information density is relatively higher for stages with higher importance; transmitting the elemental images of the plurality of stages to the second device; The second device is receiving the elemental images of the plurality of stages from the first device; and reconstructing a restored image by synthesizing the element images at a plurality of stages.

1. A communication method comprising: The first device is detecting feature information indicating the shape and position of the subject from the original image; Detecting the size of an image of a subject in a specific aspect from the original image; If the size is outside a predetermined range, the elemental image showing the subject is determined to be a basic image of the lowest level, which is the level of least importance; If the size is within the size range, the element image showing the subject is determined to be a candidate for a high-level element image that is a level of importance higher than the element image of the lowest level; The second device is reconstructing the restored image so that the morphological features are positioned at positions indicated by the feature information; When reconstructing the restored image, the element images of the plurality of levels are converted so that the information density is equal to the highest information density among the element images of the plurality of levels, and the converted element images of the plurality of levels are superimposed. Communication method.

Images