Video transmission device, line-of-sight detection device, video transmission system, video transmission method, and recording medium

Abstract

Provided are a video transmission device, a line-of-sight detection device, a video transmission system, a video transmission method, and a recording medium which are capable of dynamically adjusting the size of a high-image-quality region of a video to be transmitted.　This video transmission device is provided with: a reception unit that receives, from a line-of-sight detection device disposed on a display device side that displays a received video, information for identifying a region at which a person viewing the displayed video is gazing; a determination unit that determines, on the basis of the information, a region in the video at which the person is gazing; an adjustment unit that, on the basis of a prescribed rule, adjusts a range in the video corresponding to the region; an encoding unit that encodes the video so that the range is higher in image quality than the other portions; and a transmission unit that transmits the encoded video to the display device.

Description

Video transmission device, line of sight detection device, video transmission system, video transmission method, and recording medium

[0001] The present invention relates to a video transmission device, a line-of-sight detection device, a video transmission system, a video transmission method, and a recording medium.

[0002] A technology for improving the image quality of only objects or important areas in video captured by a camera mounted on a mobile object, etc., is known. For example, Patent Document 1 discloses an example of a mobile object control system that can reduce delay while maintaining image quality when transmitting video from a mobile object that is the target of remote support to a remote support device. Specifically, this mobile object control system sets the importance of each of multiple segmented images so that the importance of segmented images that require the remote operator's attention is higher than the importance of segmented images that require less attention. Then, this mobile object control system encodes each of the multiple segmented images so that the image quality of the segmented images with higher importance is higher than the image quality of segmented images with lower importance, and transmits them to the remote support device on the remote operator's side. Patent Document 1 also describes setting the importance of each segmented image using line-of-sight information (LOS) that indicates the remote operator's line-of-sight direction.

[0003] JP 2023-14750 A

[0004] In the mobile object control system of Patent Document 1, an image captured by a mobile object is spatially divided into a plurality of divided images, and a process is performed to improve the image quality of the divided images that are most important among the divided images. Therefore, there is a problem in that the range in which the image quality is improved compared to other divided images is determined by this division unit.

[0005] The present disclosure aims to provide a video transmission device, a line-of-sight detection device, a video transmission system, a video transmission method, and a recording medium that are capable of dynamically adjusting the range of high-quality video to be transmitted.

[0006] According to a first aspect, there is provided a video transmission device comprising: a receiving unit that receives information for identifying an area that a person watching the displayed video is gazing at from a gaze detection device arranged on the display device side that displays the received video; a determining unit that determines an area in the video that the person is gazing at based on the information; an adjusting unit that adjusts a range in the video that corresponds to the area based on a predetermined rule; an encoding unit that encodes the video so that the range has higher image quality than other parts; and a transmitting unit that transmits the encoded video to the display device.

[0007] According to a second aspect, a gaze detection device is provided that transmits information to the above-mentioned video transmission device to identify the area on which a person watching the displayed video is gazing.

[0008] According to a third aspect, there is provided a video transmission system including a video transmission device including: a receiving unit that receives information for identifying an area that a person watching the displayed video is gazing at from a gaze detection device arranged on the display device side that displays the received video; a determination unit that determines an area in the video that the person is gazing at based on the information; an adjustment unit that adjusts a range in the video that corresponds to the area based on a predetermined rule; an encoding unit that encodes the video so that the range has higher image quality than other parts; and a transmitting unit that transmits the encoded video to the display device; and a gaze detection device that transmits information to the video transmission device for identifying the area that the person watching the displayed video is gazing at.

[0009] According to a fourth aspect, there is provided a video transmission method that receives information for identifying an area that a person watching the displayed video is gazing at from a gaze detection device arranged on the display device that displays the received video, determines the area in the video that the person is gazing at based on the information, adjusts a range in the video that corresponds to the area based on predetermined rules, encodes the video so that the area has higher image quality than other parts, and transmits the encoded video to the display device.

[0010] According to a fifth aspect, there is provided a recording medium having recorded thereon a program that causes a computer to execute the following processes: receiving information from a gaze detection device arranged on a display device that displays the received video to identify an area that a person watching the displayed video is gazing at; determining an area in the video that the person is gazing at based on the information; adjusting a range in the video that corresponds to the area based on predetermined rules; encoding the video so that the area has higher image quality than other parts; and transmitting the encoded video to the display device.

[0011] According to the present disclosure, it is possible to provide a video transmission device, a line-of-sight detection device, a video transmission system, a video transmission method, and a recording medium that can dynamically adjust the range of high-quality video to be transmitted.

[0012] 1 is a diagram illustrating one configuration of the present disclosure. FIG. 1 is a flow chart illustrating the operation of the present disclosure. FIG. 2 is a diagram illustrating the operation of the present disclosure. FIG. 3 is a diagram illustrating the operation of the present disclosure. FIG. 4 is a diagram illustrating one configuration of the present disclosure. FIG. 5 is a diagram illustrating an example configuration of a gaze detection device of the present disclosure. FIG. 6 is a diagram illustrating a gaze detection method by a gaze detection device. FIG. 7 is a diagram illustrating the operation (determination of ROI) of the present disclosure. FIG. 8 is a diagram illustrating the operation (adjustment of ROI) of the present disclosure. FIG. 9 is another diagram illustrating the operation (adjustment of ROI) of the present disclosure. FIG. 10 is a diagram illustrating an example of an image displayed on a display device of the present disclosure. FIG. 11 is a diagram illustrating another configuration of the present disclosure. FIG. 12 is a diagram illustrating the operation of another configuration of the present disclosure. FIG. 13 is a diagram illustrating another configuration of the present disclosure. FIG. 14 is a diagram illustrating the operation of another configuration of the present disclosure. FIG. 15 is a diagram illustrating another configuration of the present disclosure. FIG. 16 is a diagram illustrating the operation of another configuration of the present disclosure. FIG. 17 is a diagram illustrating the configuration of a computer constituting an information processing device of the present disclosure.

[0013] First, an overview of one embodiment of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. The reference numerals in the drawings attached to this overview are attached to each element for convenience as an example to facilitate understanding, and are not intended to limit the present disclosure to the illustrated form. Furthermore, connecting lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of main signals (data) and do not exclude bidirectionality. A program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device. Furthermore, this computer device is configured to be able to communicate with internal or external devices (including computers) via the communication interface, whether wired or wireless. Although ports or interfaces are present at the input / output connection points of each block in the drawings, they are not shown.

[0014] 1 , the present disclosure can be realized in an embodiment by a configuration including a video transmission device 10 to which video captured by a camera C is input, and a line-of-sight detection device 20. More specifically, the video transmission device 10 includes a receiving unit 11, a determining unit 12, an adjusting unit 13, an encoding unit 14, and a transmitting unit 15.

[0015] The receiving unit 11 receives information for identifying an area where a person viewing the displayed video is gazing from a gaze detection device arranged on the display device that displays the received video. The determining unit 12 determines an area in the video where the person is gazing based on the information. The adjusting unit 13 adjusts a range in the video corresponding to the area based on a predetermined rule. The encoding unit 14 encodes the video so that the area has higher image quality than other parts. The transmitting unit 15 transmits the encoded video to the display device. Note that the area in the video to be adjusted may have a shape similar to that of the area in the video, or may have a shape different from that of the area in the video. In other words, adjusting the range includes at least one of adjusting (changing) the size of the area and adjusting (changing) the shape of the area.

[0016] The line-of-sight detection device 20 transmits to the video transmission device 10 information for identifying the area on which the person viewing the displayed video is gazing.

[0017] The video transmission device 10 configured as described above operates as follows: First, the video transmission device 10 receives information for identifying an area where a person viewing the displayed video is gazing from the line-of-sight detection device 20 (step S01 in FIG. 2).

[0018] Next, the video transmission device 10 determines the area in the video that the person is gazing at based on the information (step S02 in FIG. 2). Furthermore, the video transmission device 10 adjusts the range in the video that corresponds to the area based on a predetermined rule (step S03 in FIG. 2).

[0019] Next, the video transmitting device 10 encodes the video so that the image quality of the range is higher than that of other parts (step S04 in FIG. 2). Finally, the video transmitting device 10 transmits the encoded video to the display device (step S05 in FIG. 2).

[0020] The video transmission device 10 configured as described above can expand and encode the region ROI1 in a video image that a person is gazing at, based on a predetermined rule, into a range ROI2 corresponding to the region ROI1, as shown in FIG. 3 . Furthermore, the video transmission device 10 can also compress and encode the region ROI1 into a range ROI2a corresponding to the region ROI1, based on a predetermined rule, as shown in FIG. 4 . The predetermined rule can be a rule that expands the range when the upper bandwidth limit (e.g., transmission bit rate) for transmitting the video is high, or a rule that shrinks the range when the upper bandwidth limit (e.g., transmission bit rate) for transmitting the video is low. The predetermined rule can also be a rule that adjusts the corresponding range based on the time of day, environment, video quality, etc., where the video is captured. For example, a rule can be used that expands the range in the evening or nighttime, when the video is generally difficult to see, and returns the range to a standard size during the day, when the video is clearer. Of course, it is also possible to adjust the range both depending on the upper limit of the bandwidth (transmission bit rate, etc.) and the time period.

[0021] [First Embodiment] Next, a first embodiment in which the present disclosure is applied to a remote monitoring operation of a vehicle or the like will be described. Fig. 5 is a diagram showing one configuration of the present disclosure. Referring to Fig. 5, a configuration is shown in which a video transmission device 100 and a display device 300 are connected via a mobile communication network (5G / LTE). In the following embodiment, the video transmission device 100 determines a rectangular area in which high resolution should be maintained, and adjusts the size of this area to adjust the range corresponding to the area in the video.

[0022] The camera C is a camera installed on a mobile object or a facility to be monitored. The video captured by the camera C is sent to the video transmitting device 100.

[0023] The video transmission device 100 encodes video captured by a camera C and transmits the encoded video to a remote display device 300. The video transmission device 100 includes an object detection unit 101, a region determination unit 102, an encoding unit 103, and a transmission unit 104.

[0024] The object detection unit 101 uses a previously created inference model to perform object detection processing on the video captured by camera C and detect objects. This inference model is generated so that it can detect various objects in the video by learning and tuning using previously prepared training data. The object detection method used by the object detection unit 101 is not limited to this, and for example, a method of detecting objects using a difference image or frame difference may also be used. The object detection unit 101 sends information about the area in which the detected object appears, along with the video captured by camera C, to the area determination unit 102.

[0025] The region determination unit 102 determines a region in the video captured by the camera C where high resolution should be maintained. Specifically, the region determination unit 102 of this embodiment determines a region where high resolution should be maintained using not only the region where an object is detected by the object detection unit 101 but also the line-of-sight specification information received from the line-of-sight detection device 200. Furthermore, as a region adjustment process using a predetermined rule, the region determination unit 102 adjusts the size of the region determined using the line-of-sight specification information using the bandwidth upper limit information sent from the encoding unit 103. Therefore, the region determination unit 102 combines the functions of the determination unit 12 and the adjustment unit 13 described above. The bandwidth upper limit information can be, for example, a bit rate upper limit value.

[0026] The encoding unit 103 encodes the video using an encoding method that can change the image quality for each of the aforementioned regions within a predetermined upper bandwidth limit. Specifically, the encoding unit 103 of this embodiment uses this encoding method to reduce the resolution of the other regions compared to the aforementioned region. An example of such an encoding method is H.265, which supports a region of interest (ROI). This encoding method allows for setting image quality parameters for the background and the rectangular region of interest during encoding. The encoding unit 103 also sends this upper bandwidth limit information to the region determination unit 102. The encoding method used by the encoding unit 103 is not limited to H.265, and any encoding method with equivalent functionality can be used. Examples of such encoding methods include H.264 SVC (Scalable Video Coding).

[0027] In a more preferred embodiment, the network may provide the encoding unit 103 with a predicted value of the communication quality of the mobile communication network (5G / LTE). Examples of such communication quality prediction technologies include a data prediction device that predicts the probability distribution of future communication throughput from past time-series data of communication throughput (e.g., International Publication No. 2014 / 141344). In this case, the encoding unit 103 encodes video using the predicted communication quality as a bandwidth upper limit in addition to pre-set image quality parameters. This enables smooth video distribution even in unstable communication environments. Furthermore, applying the contents of the present disclosure makes it possible to maintain the image quality of the gaze area.

[0028] The transmitting unit 104 transmits the video data encoded by the encoding unit 103 to the display device 300 .

[0029] The display device 300 is a display device for monitoring purposes that is installed on a mobile object or a facility to be monitored. The display device 300 includes a receiving unit 301 and a playback unit 302. The receiving unit 301 receives video data from the video transmitting device 100 and sends the video data to the playback unit 302.

[0030] The playback unit 302 decodes and plays back the video data received from the video transmitting device 100 .

[0031] The gaze detection device 200 is a device that detects the gaze of a monitor performing monitoring duties while looking at the above-mentioned display device 300, generates information (gaze identification information) that identifies the position in the image, and sends it to the video transmission device 100.

[0032] Fig. 6 is a diagram showing an example of the configuration of a line-of-sight detection device 200. In the example of Fig. 6, the line-of-sight detection device 200 has a housing in which multiple cameras 201 are arranged horizontally, and is placed on a display device 300.

[0033] FIG. 7 is a diagram illustrating a method for detecting a gaze by a gaze detection device. The gaze detection device 200 photographs an observer with a camera 201 and acquires a gaze vector EV from the positions of the irises of the observer's eyes E1 and E2 in the facial image. The gaze detection device 200 then identifies gaze center coordinates (two-dimensional, vertical and horizontal; see reference character CC in FIG. 8 ) on the display device 300 from this gaze vector. These gaze center coordinates (two-dimensional, vertical and horizontal) can be used as the gaze identification information described above. Note that the gaze identification information is not limited to gaze center coordinates, and may be, for example, information capable of identifying a specific area the observer is looking at.

[0034] Next, the operation of this embodiment will be described with reference to the drawings. As described above, the video captured by the camera C is displayed on the display device 300 via the line-of-sight detection device 200. The line-of-sight detection device 200 identifies the line of sight of the observer watching the video, and transmits line-of-sight identification information to the line-of-sight detection device 200.

[0035] 8 to 10 are diagrams illustrating the operation of the video transmission device 100 that has received the gaze specification information. Upon receiving the gaze specification information, the video transmission device 100 first determines a region (gaze region) ROI1 of a specified size from the gaze specification information (gaze center coordinates CC) (see FIG. 8). Note that the symbol FR in FIG. 8 indicates the video frame region. Furthermore, if the coordinates specified by the gaze specification information are not on the video frame FR, this means that the observer is looking at something other than the display device 300, and the video transmission device 100 does not need to set the region ROI1.

[0036] Next, the video transmitting device 100 adjusts the size of the determined region using the bandwidth upper limit information sent from the encoding unit 103. For example, if the value of the bandwidth upper limit information is equal to or greater than a predetermined threshold (th1), the video transmitting device 100 expands the region ROI1 of the specified size to create a region (range) ROI2 (see FIG. 9 ). In this way, it becomes possible to expand the region with high image quality when there is sufficient bandwidth. The width of the expanded region can be specified using the distance from the image center coordinates, the expansion rate relative to the region ROI1 of the specified size, etc.

[0037] Furthermore, for example, when the value of the bandwidth upper limit information is less than a predetermined threshold value (th2), the video transmission device 100 creates a region (range) ROI2a by reducing the region ROI1 of the specified size (see FIG. 10). This makes it possible to reduce the region that provides high image quality when there is limited bandwidth. The width by which the region is reduced can be specified using the distance from the image center coordinates, the reduction rate relative to the region ROI1 of the specified size, etc.

[0038] Fig. 11 is a diagram showing an example of a video encoded by the video transmission device 100 of the present disclosure and displayed on the display device 300. According to the present disclosure, it is possible to display a video in which the monitor's gaze area is set at an optimal size in addition to various objects OBJ detected by the object detection unit 101. Note that the contents of the present disclosure may also be applied to the area of ​​the object OBJ in Fig. 11, and the size of the area may be adjusted using communication quality, etc.

[0039] In the above description, an example has been given in which the gaze detection device 200 is placed on the display device 300, but the type of gaze detection device 200 is not limited to this. For example, the gaze detection device 200 may be an HMD (head-mounted display) equipped with a camera or the like that can capture the movement of the wearer's eyeballs, smart glasses, VR (Virtual Reality) goggles, or the like.

[0040] In the above description, the size of the area is adjusted using the upper bandwidth limit information as one of the communication qualities of the video, but the communication qualities that can be used for size adjustment are not limited to the upper bandwidth limit. For example, parameters such as actually measured or predicted communication throughput and received signal strength (RSSI) can also be used as the communication quality.

[0041] [Second Embodiment] Next, a second embodiment will be described, which is capable of capturing the gazes of multiple observers and setting two or more regions. FIG. 12 is a diagram illustrating another configuration of the present disclosure. The first difference from the first embodiment illustrated in FIG. 5 is that the gaze detection device 200a acquires gaze vectors of multiple observers and provides gaze identification information for each observer to the video transmission device 100a. The second difference from the first embodiment is that the region determination unit 102a of the video transmission device 100a determines multiple regions using the gaze identification information and adjusts the size of each region using upper bandwidth information. The third difference from the first embodiment is that the encoding unit 103a encodes the video using an encoding method that can change image quality so that the multiple regions have higher image quality than other regions. Since the other configurations are the same as those of FIG. 5, the following description will focus on these differences.

[0042] 13 is a diagram for explaining the operation of another configuration of the present disclosure. As described above, the gaze detection device 200a acquires the gaze vectors of multiple observers and identifies the gaze center coordinates (X marks in FIG. 13 ) on the display device 300. The gaze detection device 200a then transmits these gaze center coordinates to the video transmission device 100a as gaze identification information.

[0043] Then, the video transmitting device 100a adjusts the sizes of the multiple regions using the upper bandwidth limit information sent from the encoding unit 103. This allows the image quality of the region being watched by multiple observers to be improved.

[0044] 13 shows an example in which multiple observers are watching one display device 300, there may be two or more display devices 300. In this case, a line-of-sight detection device 200 may be provided for each display device.

[0045] As described above, the present disclosure can be applied without any problems to cases where multiple monitors are on duty.

[0046] [Third Embodiment] Next, a third embodiment will be described in which the size of the attention area is changed using video quality information instead of the upper bandwidth limit. Fig. 14 is a diagram showing another configuration of the present disclosure. The difference from the first embodiment shown in Fig. 5 is that the area determination unit 102b of the video transmission device 100b adjusts the size of the attention area using video quality information. Since the other configurations are the same as the configuration of Fig. 5, the following description will focus on the differences.

[0047] The video transmission device 100b adjusts the size of the determined region based on the quality of the video. For example, if the frame rate or resolution of the video is equal to or greater than a predetermined threshold (th1), the video transmission device 100b creates an ROI2a by reducing the region ROI1 of the specified size (see FIG. 10). Similarly, if the frame rate or resolution of the video is less than a predetermined threshold (th2), the video transmission device 100b creates an ROI2 by expanding the region ROI1 of the specified size (see FIG. 9). If the quality of the original video is high, reducing the region ROI1 of the specified size is likely to have little impact on the monitoring operation. Conversely, if the quality of the original video is low, expanding the region ROI1 of the specified size can make the monitoring operation easier.

[0048] As described above, the present disclosure can be modified to change the size of the attention area using video quality information instead of the upper bandwidth limit.

[0049] [Fourth Embodiment] Next, a fourth embodiment will be described in which the size of the attention area is changed using time instead of the upper bandwidth limit. Fig. 15 is a diagram showing another configuration of the present disclosure. The difference from the first embodiment shown in Fig. 5 is that the video transmission device 100c is provided with a clock 105, and the area determination unit 102c adjusts the size of the attention area based on the time when the video is captured. Since the other configurations are the same as those of Fig. 5, the following description will focus on the differences.

[0050] The video transmission device 100c adjusts the size of the determined region based on the current time. For example, when the time is in the morning or evening or at night, the video transmission device 100c creates ROI2, which is an expanded region ROI1 of the specified size (see FIG. 9). Conversely, when the time is daytime, the video transmission device 100c creates ROI2a, which is a reduced region ROI1 of the specified size (see FIG. 10). When the time is in the morning or evening or at night, expanding the region ROI1 of the specified size makes it easier to perform monitoring tasks. Conversely, when the time is other than that, reducing the region ROI1 of the specified size is considered to have little impact on monitoring tasks.

[0051] As described above, the present disclosure can be modified to change the size of the fixation area using time instead of the upper band limit value.

[0052] [Fifth Embodiment] Next, a fifth embodiment will be described in which the size of the gaze area is changed based on the environment in which the image capturing device is located, instead of the upper bandwidth limit. Figure 16 is a diagram showing another configuration of the present disclosure. The difference from the first embodiment shown in Figure 5 is that the video transmitting device 100d is capable of receiving sensor data from a sensor S installed near the camera C, and the area determining unit 102d adjusts the size of the gaze area based on the sensor data. Since the other configurations are the same as those of Figure 5, the following description will focus on the differences.

[0053] The video transmission device 100d adjusts the size of the determined region based on sensor data representing the environment in which the camera (image capture device) C capturing the video is located. Examples of such sensor data include illuminance sensors and weather sensors. For example, if the video transmission device 100d determines based on the sensor data that the video is dark, it creates an ROI2 by enlarging the region ROI1 of the specified size (see FIG. 9). Conversely, if the video transmission device 100c determines based on the sensor data that there is no problem with the brightness of the video, it either leaves the region ROI1 of the specified size unchanged or creates a reduced ROI2a (see FIG. 10).

[0054] 17 is a diagram illustrating the operation of another configuration of the present disclosure. For example, as shown in FIG. 17, if the location where the camera (photography device) C is placed is dark, the surveillance work can be facilitated by enlarging the specified size of region ROI1 and setting region ROI2. Conversely, if there are no problems with the environment in which the camera (photography device) is placed, the specified size of region ROI1 can be used as is or reduced, with little impact on the surveillance work.

[0055] As described above, the present disclosure can be modified to a form in which the size of the gaze area is changed using sensor data that represents the environment in which the camera (imaging device) C is placed, instead of the upper bandwidth limit value.

[0056] [Sixth Embodiment] In the fifth embodiment described above, the environment in which the image capturing device is placed is estimated using a sensor S, but a camera C can also be used instead of the sensor. Next, a sixth embodiment will be described in which the image from the camera C is used as sensor data to change the size of the gaze area. FIG. 18 is a diagram showing another configuration of the present disclosure. The sixth embodiment differs from the first embodiment shown in FIG. 5 in that a scene detection unit 106 is added to the video transmission device 100e, and the area determination unit 102e adjusts the size of the gaze area based on the output of the scene detection unit 106. Other configurations are the same as those in FIG. 5, so the following description will focus on the differences.

[0057] In the configuration of FIG. 18 , the video captured by camera C is also input to the scene detection unit 106. The scene detection unit 106 detects brightness and scene from the video captured by camera C. Note that brightness can be measured from the average brightness value of the video sent from camera C, etc. Furthermore, the scene can be detected as "sunny," "cloudy," "rainy," etc. using various automatic scene determination technologies. For example, as shown in FIG. 17 , if the location where camera (camera) C is located is dark as a result of the brightness and scene detection, the surveillance work can be facilitated by enlarging the region ROI1 of the specified size. Conversely, if there are no problems with the environment in which the camera (camera) is located, the impact on the surveillance work is thought to be small, even if the region ROI1 of the specified size is used as is or reduced.

[0058] As described above, the present disclosure can be modified to a form in which the size of the fixation area is changed using an image from a camera (photographing device) C instead of the sensor S.

[0059] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of the present disclosure. For example, the network configurations, element configurations, and data representation formats shown in the drawings are examples intended to aid in understanding the present disclosure, and are not limited to the configurations shown in these drawings.

[0060] For example, in the first to sixth embodiments described above, the video transmission devices 100 to 100e and the display device 300 are described as being connected via a mobile communication network (5G / LTE), but they may be connected via a network other than a mobile communication network. For example, the present disclosure can be applied without any problems even when the video transmission devices 100 to 100e and the display device 300 are connected via the Internet or a dedicated line.

[0061] Furthermore, for example, in the above-described embodiment, an example was given in which a rectangular gaze area was identified from the observer's line of sight and its size was changed, but the range in which high image quality is maintained does not necessarily have to be rectangular. For example, based on the rectangular gaze area, a range of any shape and size that maintains high image quality compared to other areas can be set, and the range can be encoded to achieve high image quality. Such a range of any shape can be set by combining multiple rectangles.

[0062] (Regarding Hardware Configuration) In each embodiment of the present disclosure, each component of each device represents a functional unit block. Some or all of the components of each device are realized by an arbitrary combination of an information processing device 900 and a program, for example, as shown in FIG. 19 . FIG. 19 is a block diagram showing an example of the hardware configuration of the information processing device 900 that realizes each component of each device. The information processing device 900 includes, as an example, the following configuration: - CPU (Central Processing Unit) 901 - ROM (Read Only Memory) 902 - RAM (Random Access Memory) 903 - Program 904 loaded into RAM 903 - Storage device 905 that stores the program 904 - Drive device 907 that reads and writes to a recording medium 906 - Communication interface 908 that connects to a communication network 909 - Input / output interface 910 that inputs and outputs data - Bus 911 that connects each component

[0063] Each component of each device in each embodiment is realized by the CPU 901 acquiring and executing a program 904 that realizes these functions. That is, the CPU 901 in FIG. 19 executes an area determination program and a size adjustment program, and performs an update process for each calculation parameter stored in the RAM 903, the storage device 905, etc. The program 904 that realizes the function of each component of each device is stored in the storage device 905 or the ROM 902 in advance, for example, and is read by the CPU 901 as needed. The program 904 may be supplied to the CPU 901 via the communication network 909, or may be stored in advance on the recording medium 906, and the drive device 907 may read the program and supply it to the CPU 901.

[0064] There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a separate information processing device 900 and a program for each component. Furthermore, multiple components of each device may be realized by any combination of a single information processing device 900 and a program. That is, each unit (processing means, function) of the video transmission device shown in the first to sixth embodiments can be realized by a computer program that causes a processor installed in the device to execute each of the above-mentioned processes using its hardware.

[0065] In addition, some or all of the components of each device may be realized by other general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or by multiple chips connected via a bus.

[0066] Some or all of the components of each device may be realized by a combination of the above-mentioned circuits and programs.

[0067] When some or all of the components of each device are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in a form in which each device is connected via a communication network.

[0068] It should be noted that the above-described embodiments are preferred embodiments of the present disclosure, and the scope of the present disclosure is not limited to only the above-described embodiments. In other words, those skilled in the art can modify or substitute the above-described embodiments to construct various modified forms without departing from the gist of the present disclosure.

[0069] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0070] [Supplementary Note 1] A video transmission device comprising: a receiving unit that receives information for identifying an area that a person viewing the displayed video is gazing at from a line-of-sight detection device arranged on a display device that displays the received video; a determining unit that determines an area in the video that the person is gazing at based on the information; an adjusting unit that adjusts a range in the video corresponding to the area based on a predetermined rule; an encoding unit that encodes the video so that the area has higher image quality than other parts; and a transmitting unit that transmits the encoded video to the display device. [Supplementary Note 2] The adjusting unit of the video transmission device may be configured to adjust the range based on communication quality of the video. [Supplementary Note 3] The adjusting unit of the video transmission device may be configured to adjust the range based on video quality. [Supplementary Note 4] The adjusting unit of the video transmission device may be configured to adjust the range based on the time the video is captured. [Supplementary Note 5] The adjusting unit of the video transmission device may be configured to adjust the range based on sensor data indicating an environment in which a capture device that captures the video is located. [Supplementary Note 6] The above-mentioned video transmission device may further include an object detection unit that detects a predetermined object in the video, and the encoding unit may encode the video so that the range and the area where the object is displayed have higher image quality than other areas. [Supplementary Note 7] The receiving unit of the above-mentioned video transmission device may receive information from the line-of-sight detection device for identifying areas where multiple people viewing the displayed video are gazing, and the determining unit may determine areas in the video where each person is gazing based on the information. [Supplementary Note 8] The encoding unit of the above-mentioned video transmission device may be configured to use an encoding method that reduces the resolution of the other areas compared to the range. [Supplementary Note 9] The encoding unit of the above-mentioned video transmission device may be configured to perform encoding based on a predicted value of the communication quality provided from an external device. [Supplementary Note 10] A line-of-sight detection device may transmit information to the above-mentioned video transmission device for identifying areas where people viewing the displayed video are gazing.[Supplementary Note 11] A video transmission system including: a video transmission device comprising: a receiving unit that receives, from a gaze detection device arranged on a display device that displays received video, information for identifying an area that a person viewing the displayed video is gazing at, a determining unit that determines an area in the video that the person is gazing at based on the information, an adjusting unit that adjusts a range in the video that corresponds to the area based on a predetermined rule, an encoding unit that encodes the video so that the area has a higher image quality than other parts, and a transmitting unit that transmits the encoded video to the display device; and a gaze detection device that transmits, to the video transmission device, information for identifying an area that the person viewing the displayed video is gazing at. [Supplementary Note 12] A video transmission method comprising: receiving, from a gaze detection device arranged on a display device that displays received video, information for identifying an area that a person viewing the displayed video is gazing at, determining an area in the video that the person is gazing at based on the information, adjusting a range in the video that corresponds to the area based on a predetermined rule, encoding the video so that the area has a higher image quality than other parts, and transmitting the encoded video to the display device. [Supplementary Note 13] A recording medium storing a program that causes a computer to execute the following steps: receiving information for identifying an area where a person viewing the displayed video is gazing from a gaze detection device disposed on a display device that displays the received video; determining an area in the video where the person is gazing based on the information; adjusting a range in the video corresponding to the area based on a predetermined rule; encoding the video so that the area has higher image quality than other parts; and transmitting the encoded video to the display device. Note that the embodiments described in each of the above supplementary notes can be combined with each other after making necessary modifications. For example, a configuration that combines the contents of Supplementary Note 2 and Supplementary Note 3 and adjusts the size of the area using both communication quality and video quality is also within the scope of the disclosure of this specification.The embodiments of Supplementary Notes 11 to 13 can be expanded into the embodiments of Supplementary Notes 2 to 8, similarly to Supplementary Note 1.

[0071] The disclosures of the above-cited patent documents are incorporated herein by reference and may be used as the basis or part of this disclosure, as necessary. Modifications and adjustments of the embodiments and examples are possible within the scope of this disclosure (including the claims), and further based on its basic technical concept. Furthermore, various combinations and selections (including partial deletions) of various disclosed elements (including elements of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of this disclosure. In other words, this disclosure naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure, including the claims, and the technical concept. In particular, with regard to the numerical ranges described herein, any numerical value or subrange within that range should be construed as specifically described, even if not otherwise specified. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of this disclosure, in accordance with the spirit of this disclosure, are also deemed to be included in the disclosures of this application.

[0072] 10 Video transmission device 11 Receiving unit 12 Determining unit 13 Adjusting unit 14 Encoding unit 15 Transmitting unit 20 Line of sight detection device 100, 100a to 100e Video transmission device 101 Object detection unit 102, 102a to 102e Area determination unit 103, 103a Encoding unit 104 Transmitting unit 105 Clock 106 Scene detection unit 200 Line of sight detection device 201 Camera 300 Display device 301 Receiving unit 302 Playback unit 900 Information processing device 901 CPU (Central Processing Unit) 902 ROM (Read Only Memory) 903 RAM (Random Access Memory) 904 Program 905 Storage device 906 Recording medium 907 Drive device 908 Communication interface 909 Communication network 910 Input / output interface 911 Bus C Camera FR Image frame area CC Line of sight center coordinates E1, E2 Surveillance officer's eyes EV Line of sight vector FR Image frame area OBJ Object S Sensor ROI1 Area ROI2, ROI2a Range

Claims

1. A video transmission device comprising: a receiving unit that receives, from a line-of-sight detection device disposed on the display device side for displaying the received video, information for specifying an area being gazed at by a person watching the displayed video; a determining unit that determines, based on the information, an area in the video being gazed at by the person; an adjusting unit that adjusts, based on a predetermined rule, a range in the video corresponding to the area; an encoding unit that encodes the video so that the range has higher image quality compared to other parts; and a transmitting unit that transmits the encoded video to the display device.

2. The video transmission device according to claim 1, wherein the adjusting unit adjusts the range based on the communication quality of the video.

3. The video transmission device according to claim 1 or 2, wherein the adjusting unit adjusts the range based on the quality of the video.

4. The video transmission device according to any one of claims 1 to 3, wherein the adjusting unit adjusts the range based on the time when the video is being shot.

5. The video transmission device according to any one of claims 1 to 4, wherein the adjusting unit adjusts the range based on sensor data indicating the environment where the shooting device shooting the video is placed.

6. Further comprising an object detection unit that detects a predetermined object shown in the video, wherein the encoding unit encodes the video so that the range and the area where the object is shown have higher image quality compared to other parts, the video transmission device according to any one of claims 1 to 5.

7. The receiving unit receives, from the line-of-sight detection device, information for specifying areas being gazed at by a plurality of persons watching the displayed video, and the determining unit determines, based on each of the information, areas in the video being gazed at by the respective persons, the video transmission device according to any one of claims 1 to 6.

8. The video transmission device according to any one of claims 1 to 7, wherein the encoding unit uses an encoding method that reduces the resolution of the other parts more than the range.

9. The video transmission device according to any one of claims 1 to 8, wherein the encoding unit performs encoding based on a predicted value of the communication quality provided from the outside.

10. A line-of-sight detection device that transmits information for specifying an area being gazed at by a person watching the displayed video to the video transmission device according to any one of claims 1 to 9.

11. A video transmission system, comprising: a receiving unit that receives, from a line-of-sight detection device disposed on the display device side for displaying the received video, information for specifying a region that a person watching the displayed video is gazing at; a determination unit that determines, based on the information, the region in the video that the person is gazing at; an adjustment unit that adjusts, based on a predetermined rule, a range in the video corresponding to the region; an encoding unit that encodes the video so that the range has higher image quality compared to other parts; and a transmission unit that transmits the encoded video to the display device; and a line-of-sight detection device that transmits, to the video transmission device, information for specifying a region that a person watching the displayed video is gazing at.

12. A video transmission method, comprising: receiving, from a line-of-sight detection device disposed on the display device side for displaying the received video, information for specifying a region that a person watching the displayed video is gazing at; determining, based on the information, the region in the video that the person is gazing at; adjusting, based on a predetermined rule, a range in the video corresponding to the region; encoding the video so that the range has higher image quality compared to other parts; and transmitting the encoded video to the display device.

13. A recording medium having recorded thereon a program for causing a computer to execute: a process of receiving, from a line-of-sight detection device disposed on the display device side for displaying the received video, information for specifying a region that a person watching the displayed video is gazing at; a process of determining, based on the information, the region in the video that the person is gazing at; a process of adjusting, based on a predetermined rule, a range in the video corresponding to the region; a process of encoding the video so that the range has higher image quality compared to other parts; and a process of transmitting the encoded video to the display device.

Images