Conversion processing device and image transmission system

Abstract

The present invention makes it possible to further reduce a bit rate while suppressing and compensating an information loss due to quality degradation in video data and image data.　A conversion processing device 20 disposed on the transmission side comprises: an object detection unit 21 that detects an object from an image captured by an imaging device 10 and generates object information; a resizing unit 23 that compares the object information with a predetermined threshold to determine whether or not the degree of an adverse effect due to an information loss when the image is reduced exceeds a standard, if the degree of the adverse effect due to the information loss is determined to exceed the standard, does not reduce the image, and if not, reduces the image; and an encoding unit 24 that encodes the image processed by the resizing unit 23.

Description

Conversion processing device and image transmission system

[0001] The present invention relates to a conversion processing device that converts images captured by an image capturing device into images suitable for transmission, and to an image transmission system that uses the same.

[0002] Currently, various wireless communication standards have been formulated, differing in factors such as the frequency bands used. Among these, a wireless communication technology known as LPWA (Low Power Wide Area) has advantages in terms of transmission distance and power consumption, despite its relatively slow transmission speed. Specific LPWA standards include LoRaWAN and NB-IoT. Due to its characteristics, LPWA is often used in applications such as IoT (Internet of Things), for example, for transmitting relatively simple data, such as transmitting and receiving numerical data acquired by various sensors and transmitting and receiving control signals for devices.

[0003] Wi-Fi HaLow (IEEE 802.11ah) is a wireless communication standard that became available for domestic use in recent years (September 2022) following amendments to the Radio Law. Wi-Fi HaLow is a Wi-Fi (registered trademark) (IEEE 802.11) standard that allows for particularly long-distance communication. It has characteristics similar to LPWA, but boasts faster transmission speeds than LPWA. Given its features, Wi-Fi HaLow is expected to be used to bridge the gap between traditional Wi-Fi and LPWA. A typical example is the long-distance transmission of video and images captured by cameras, which could be used in a variety of fields and situations, including farm and field management, river and public facility monitoring, and video analysis in retail and distribution.

[0004] Meanwhile, in recent years, there has been remarkable development of AI (Artificial Intelligence) technology based on machine learning both in Japan and overseas. The development of AI technology has made it possible to perform advanced information processing of unstructured data, which was difficult with conventional computer programming techniques, and the value of video data and image data is increasing.

[0005] For example, it has become possible to achieve with high accuracy image processing tasks such as "object detection" (e.g., Non-Patent Document 1), which is a task of estimating the position and type of an object in an image, "super-resolution" (e.g., Non-Patent Document 2), which is a task of restoring a high-resolution image from a low-resolution image, and "frame interpolation" (e.g., Non-Patent Document 3), which is a task of interpolating an intermediate frame from two temporally close frames. Furthermore, by combining it with different modalities such as natural language, more flexible information processing is becoming possible, such as question answering for images and videos (e.g., Non-Patent Document 4).

[0006] Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” June 8, 2015, [online], https: / / arxiv.org / abs / 1506.02640.Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang, “Image Super-Resolution Using Deep Convolutional Networks,” December 31, 2014, [online], https: / / arxiv.org / abs / 1501.00092.Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, Shuchang Zhou, “Real-Time Intermediate Flow Estimation for Video Frame Interpolation,” November 12, 2020, [online], https: / / arxiv.org / abs / 2011.06294.Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, Devi Parikh, “VQA: Visual Question Answering,” May 3, 2015, [online], https: / / arxiv.org / abs / 1505.00468.

[0007] When transmitting video or image data, such as from a live camera, via wireless communication, the quality of the transmittable data primarily depends on the transmission speed of the wireless network. While several factors determine the quality of video and images, the primary factor is resolution (number of pixels). Furthermore, video and images are typically transmitted using a codec for data compression, and the compression rate also affects quality. For example, lossy codecs, such as the H.264 video codec and the JPEG image codec, have a trade-off between compression rate and image quality; increasing the compression rate can adversely affect image quality, such as increasing block noise.

[0008] Furthermore, when focusing on information in the time domain as well as information in the pixel space, the frame rate (the number of frames transmitted per second) also affects the total amount of data transmitted. By reducing the frame rate, the data volume can be reduced, but the amount of temporal information is lost.

[0009] Based on the above, in this specification, we will treat the three elements of "resolution," "image quality (how an image appears, which changes mainly due to lossy compression codecs)," and "frame rate" as factors that determine the quality of video or images.

[0010] Generally, wireless communication has a transmission speed limit, so the quality of the video or image to be transmitted must be within the available bit rate. In particular, with communication standards such as Wi-Fi HaLow, in practice, quality must often be sacrificed due to factors such as the standard and the performance of the communication equipment, as well as the transmission time rate restrictions set by the Radio Law. This creates a trade-off between resolution, image quality, and frame rate.

[0011] Regarding resolution, even if the resolution is reduced during data transmission, it is possible to restore an image close to the original by using known super-resolution techniques after data transmission. However, in principle, it is impossible to restore the original image exactly. When the resolution is reduced, pixel restoration errors and information loss can occur due to various factors such as the resizing ratio and the amount of information in the original pixel space.

[0012] Regarding image quality, the trade-off between image quality and compression rate can be improved by adopting a codec with better compression efficiency. However, when comparing with the same codec, increasing the compression rate results in a significant degradation of image quality and loss of detailed information. Specific examples of adverse effects include a decrease in the reproducibility of areas with large movements in video compression, and a decrease in the reproducibility of high-frequency information such as the contours of objects in image compression.

[0013] Regarding frame rate, even if frames are thinned out during data transmission to reduce the frame rate, it is possible to obtain an image close to the thinned frames by using known frame interpolation techniques after data transmission. However, it is theoretically impossible to accurately interpolate the thinned frames. In general, the greater the difference between the frames used for frame interpolation, the lower the interpolation accuracy. In addition, it is technically difficult to interpolate objects or movements that are captured momentarily only in the thinned frames.

[0014] As described above, it is desirable to reduce the bit rate when transmitting video data or image data via wireless communication, but sacrificing any of the resolution, image quality, and frame rate can result in information loss in the video or image. The present invention has been made in consideration of the above-mentioned conventional circumstances, and aims to further reduce the bit rate while suppressing and compensating for information loss due to deterioration in the quality of video data or image data.

[0015] An image conversion device according to one aspect of the present invention is a conversion processing device that converts an image captured by an imaging device for transmission, and includes an object detection unit that detects an object from the image and generates object information, a resizing unit that compares the object information with a predetermined threshold to determine whether the degree of adverse impact due to information loss when the image is reduced exceeds a standard, and does not reduce the image if it is determined that the degree of adverse impact due to information loss exceeds the standard, but reduces the image otherwise, and an encoding unit that encodes the image that has been processed by the resizing unit.

[0016] Here, in the above conversion processing device, the object detection unit may detect objects of a plurality of different classes, and the threshold value may be set for each class.

[0017] In addition, in the above-mentioned conversion processing device, the object information includes a bounding box that indicates the position and size of an object detected from the image using a rectangle, and the predetermined threshold is a threshold related to the area of ​​the bounding box, and the resizing unit may determine that the degree of adverse effect due to information loss exceeds a standard if the area of ​​the smallest bounding box among one or more bounding boxes included in the object information is smaller than the predetermined threshold and not reduce the image, but may reduce the image if this is not the case.

[0018] In addition, in the above-mentioned conversion processing device, the object information includes a score representing a degree of certainty regarding an object detected from the image, the predetermined threshold is a threshold regarding the score, and the resizing unit may determine that the degree of adverse effect due to information loss exceeds a standard if the smallest score among one or more scores included in the object information is smaller than the predetermined threshold and not reduce the image, but may reduce the image if this is not the case.

[0019] In the above conversion processing device, the resizing section may determine a magnification for reducing the image in accordance with the object information.

[0020] In the above conversion processing device, when the image is reduced by the resizing section, the encoding section may embed information about the resolution before reduction as metadata of the reduced image.

[0021] Another aspect of the present invention is an image transmission system that transmits an image captured by an imaging device from a transmitting side to a receiving side, wherein the conversion processing device deployed on the transmitting side comprises an object detection unit that detects an object from the image and generates object information, a resizing unit that compares the object information with a predetermined threshold to determine whether the degree of adverse impact due to information loss when the image is reduced exceeds a standard, and does not reduce the image if it is determined that the degree of adverse impact due to information loss exceeds the standard, but reduces the image otherwise, and an encoding unit that encodes the image that has been processed by the resizing unit; and the restoration processing device deployed on the receiving side comprises a decoding unit that decodes data received from the transmitting side, and a restoration unit that, if the image decoded by the decoding unit has been reduced, restores the image to the resolution before reduction.

[0022] According to the present invention, it is possible to further reduce the bit rate while suppressing and compensating for information loss due to quality degradation of video data or image data, thereby enabling efficient data transmission even in wireless communication where the transmission speed is limited.

[0023] FIG. 1 is a diagram showing an overview of an image transmission system according to an embodiment of the present invention. FIG. 2 is a diagram showing an overview of processing by a conversion processing device of the image transmission system of FIG. 1. FIG. 3 is a diagram showing an overview of processing by a restoration processing device of the image transmission system of FIG. 1. FIG. 4 is a diagram showing an example of the hardware configuration of the conversion processing device of FIG. 2. FIG. 5 is a diagram showing an example of the hardware configuration of the restoration processing device of FIG. 3. FIG. 6 is a diagram showing an example of an algorithm of processing executed by the conversion processing device of FIG. 3. FIG. 7 is a diagram showing an example of how an image appears at each stage of image transmission using a conventional method. FIG. 8 is a diagram showing an example of how an image appears at each stage of image transmission when the algorithm of FIG. 6 is executed. FIG. 9 is a diagram showing an example of how an image appears at each stage of image transmission when the algorithm of FIG. 6 is executed.

[0024] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following description is an example, and the present invention is not limited thereto. Fig. 1 shows an overview of an image transmission system according to an embodiment of the present invention. Arrows in the figure represent the flow of data between devices, with solid arrows representing wired data transmission and dashed arrows representing wireless data transmission.

[0025] The image transmission system of Figure 1 is provided on the sending side with an imaging device 10 that captures images, a conversion processing device 20 that analyzes the captured images and performs data conversion, and a communication device (transmitter) 30 that transmits image data via wireless communication, and on the receiving side with a communication device (receiver) 40 that receives image data via wireless communication and a restoration processing device 50 that restores the captured image from the received image data.

[0026] The image capturing device 10 is a camera capable of capturing video and images, such as a network camera or web camera. By connecting the image capturing device 10 and the conversion processing device 20 via Ethernet, USB (Universal Serial Bus), or the like, the conversion processing device 20 can capture images from the image capturing device 10 in real time.

[0027] As shown in Fig. 2, the conversion processing device 20 includes an image acquisition unit 21 that acquires an image from the imaging device 10, an object detection unit 22 that detects an object from the image and generates object information, a resizing unit 23 that resizes the image based on the object information, and an encoding unit 24 that encodes the image. As shown in Fig. 4, the conversion processing device 20 has, as hardware, one or more processors 201 that execute these processes and one or more main memories 202 that store programs and data required for these processes. The processor 201 can be freely selected from existing processors such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and two or more of these can also be used in any combination.

[0028] The communication devices (30, 40) are composed of a transmitter 30 for transmitting data encoded by the conversion processing device 20 and a receiver 40 for receiving data from the transmitter 30. Regarding the wireless communication performed between the transmitter 30 and the receiver 40, any communication method can be used, and a method can be freely selected from existing wireless standards, taking into account factors such as the convenience of procuring communication equipment and the desired wireless band. However, the present invention is expected to be particularly effective with standards such as Wi-Fi HaLow (IEEE 802.11ah).

[0029] As shown in FIG. 3 , the restoration processing device 50 includes a decoding unit 51 that decodes received data, a restoration unit 52 that restores the decoded image to its original size if the image has been reduced, and a storage unit 53 that stores the restored image. The restoration processing device 50 includes, as hardware, one or more processors 501 for executing these processes, one or more main memories 502 for storing programs and data required for these processes, and one or more storages 503 for recording data generated as a result of these processes. Like the processor 201, the processor 501 can be selected and configured arbitrarily. The storage 503 can be freely selected from existing devices such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0030] The transmitting devices (i.e., the image capture device 10, the conversion processing device 20, and the communication device (transmitter) 30) shown in FIG. 1 need to be installed close to each other. Similarly, the receiving devices (i.e., the communication device (receiver) 40 and the restoration processing device 50) need to be installed close to each other. In contrast, the transmitting and receiving devices may be installed as far apart as possible. Furthermore, if a sufficient data transmission speed can be ensured, the transmitting or receiving devices may be connected wirelessly using Wi-Fi or the like instead of being connected by wire.

[0031] An overview of the processing performed by the conversion processing device 20 will be described with reference to Figure 2. The conversion processing device 20 first acquires an image from the image capture device 10 using the image acquisition unit 21. The image acquired here may be a still image captured by the image capture device 10 each time, or, if the image capture device 10 is a video camera, a single frame from a video being captured. Next, the conversion processing device 20 performs object detection processing on the image acquired from the image capture device 10 using the object detection unit 22, and generates object information indicating the results. The object detection processing method can be freely selected from known methods, and for example, YOLO (see Non-Patent Document 1) can be applied.

[0032] Next, the conversion processing device 20 can resize (reduce) the image using the resizing unit 23. Whether or not resizing is applicable is determined by analyzing object information obtained by the object detection process in the object detection unit 22. If the analysis of the object information determines that the information of interest is unlikely to be lost even if the image is reduced, or if it determines that any loss of information can be sufficiently compensated for by restoration processing, the image is reduced; otherwise, the image is left at the same size without being reduced. This makes it possible to reduce the data volume of the image.

[0033] Next, the conversion processing device 20 encodes the image output from the resizing unit 23 using the encoding unit 24. The codec and encoder used for encoding can be freely selected from existing ones. The encoded data is wirelessly transmitted to the receiving side by the subsequent communication device (transmitter) 30. The conversion processing device 20 can arbitrarily repeat the above-mentioned processing. For example, still images can be processed intermittently every few seconds to several minutes, or a video stream can be processed in real time.

[0034] An overview of the processing performed by the restoration processing device 50 will be described with reference to FIG. 3 . The restoration processing device 50 first decodes data received by the communication device (receiver) 40 using a decoding unit 51. Next, if the restoration processing device 50 determines that the decoded image has been reduced, the restoration unit 52 restores the image to its original resolution. The restoration method can be freely selected from known methods depending on factors such as restoration quality and computational complexity, such as image enlargement using a resampling algorithm (e.g., bilinear or Lanczos algorithm) or super-resolution using super-resolution AI (see Non-Patent Document 2). Next, the restoration processing device 50 stores the restored image in storage using a storage unit 53.

[0035] In the present invention, how to determine whether to reduce an image in the resizing unit 23 of the conversion processing device 20 shown in Fig. 2 is particularly important. The processing executed by the conversion processing device 20 will be described with reference to Fig. 6. Here, a method will be described in which the size of the bounding box (a rectangle indicating the position and size of an object in an image) among the object information indicating the object detection result is focused on to evaluate the impact of information loss that occurs when the image is reduced, and to determine whether to reduce the image.

[0036] 6, the conversion processing device 20 first sets a threshold value θ for the area of ​​the bounding box that serves as the evaluation standard (step S11). The threshold value θ is set, for example, as a preliminary step before starting operation of the present system, but the threshold value θ may also be changed as appropriate during operation.

[0037] Thereafter, the image acquisition unit 21 acquires an image from the image capture device 10 (step S12), and the object detection unit 22 performs object detection on the acquired image to acquire a bounding box as the object detection result (step S13). Here, the class of detection target (type of object to be detected) is arbitrary, but it is desirable to detect enough objects to satisfy the final use of the captured image. For example, if the purpose is to record and analyze the movements and states of people, people should be the detection target, and if the purpose is to record and analyze traffic conditions on roads and parking lots, passenger cars, bicycles, etc. should be the detection target.

[0038] Next, the resizing unit 23 determines whether or not a bounding box exists as a result of the object detection (step S14). If one or more bounding boxes exist, the resizing unit 23 selects the bounding box B having the smallest area among the bounding boxes. min (Step S15), and the evaluation formula (B min It is determined whether the bounding box (area of ​​the image) > θ is true or false (step S16). If step S16 is true, or if it is determined in step S14 that a bounding box does not exist, it is estimated that the degree of information loss that occurs when the image is reduced is small, so the image is reduced (step S17).

[0039] Next, the image is encoded by the encoding unit 24 (step S18). The encoded image data is transmitted by the communication device (transmitter) 30 to the communication device (receiver) 40 (step S19). After that, if the processing is to be continued, the procedures of steps S12 to S19 are repeated; if not, the processing is terminated (step S20).

[0040] The effects expected from executing the above-described algorithm will be described with reference to FIGS.

[0041] FIG. 7 shows examples of how an image appears at each stage when a captured image is simply reduced and then restored to its original resolution by enlargement processing. FIG. 7(a) shows an image captured by the image capture device 10. The image shows a car and two people, one of whom appears relatively large because he is walking in the foreground, while the other appears relatively small because he is standing upright in the distance. FIG. 7(b) shows an image obtained by reducing the image in FIG. 7(a), and the dashed rectangle indicates the size of the image before reduction. FIG. 7(c) shows an image obtained by enlarging the image in FIG. 7(b) and restoring it to its original resolution.

[0042] Focusing on the individual objects in the image of Figure 7(c), the car and the person in the foreground appear jagged and blurred in their contours compared to the image of Figure 7(a), but their general shapes are still reproduced. In other words, the image of Figure 7(c) retains all information about the car's windows and tires, the person's head and limbs, and other details, making them easily recognizable at a glance. Meanwhile, among the objects in the image of Figure 7(c), the person in the distance has lost detailed information due to the reduction in size, making it difficult to distinguish the contours of his or her limbs even when restored to the original resolution.

[0043] In this way, even if an image is reduced by a certain factor, the extent of the adverse effect of information loss due to reduction depends on the appearance of the object of focus in the image. In particular, as observed in this example, when the object of focus appears smaller, it is expected that the adverse effect of reduction will be more pronounced.

[0044] 8 and 9 show examples of how an image appears at each stage when the algorithm shown in FIG. 6 is executed. FIG. 8 shows an example of executing the above-mentioned algorithm on the same captured image as FIG. 7. FIG. 8(a) shows the image acquired in step S12. FIG. 8(b) shows the image of FIG. 8(a) with the bounding box acquired in step S13 superimposed on it. The bounding box is displayed so as to surround each object in the image, and the class determined by object detection and the confidence score are displayed above the bounding box. The confidence score is a numerical representation of how likely the detected object is to belong to that class.

[0045] In this case, the bounding box of the person in the distance has the smallest area, so the size of this bounding box area is compared with the threshold value θ in step S16. Figure 8(c) shows how the transmitted image appears in step S19 when step S16 is evaluated as false. In this case, the reduction process in step S17 is skipped, so although the data volume cannot be reduced by reducing the image, the image data is transmitted with the same clear appearance as in Figure 8(a).

[0046] 9A and 9B show an example of the above-described algorithm being applied to a captured image in which no people are visible in the distance. Fig. 9A shows the image acquired in step S12. Fig. 9B shows the image in Fig. 9A with the bounding box acquired in step S13 superimposed thereon.

[0047] In this case, the area of ​​the person's bounding box is smallest, so the size of this bounding box area is compared with the threshold value θ in step S16. Figure 9(c) shows how the transmitted image appears in step S19 when step S16 is evaluated as true. In this case, the image is reduced in step S17, resulting in a reduction in the amount of transmitted data.

[0048] 9(d) shows an image restored to the original resolution by the restoration processing device 50. Although the quality of the cars and people has deteriorated, the general shapes of the cars and people have been reproduced. In this example, it can be said that the restoration process has compensated for the loss of resolution due to image reduction, while also reducing the amount of transmitted data.

[0049] As described above, the image transmission system of this example includes a conversion processing device 20 on the transmitting side, an object detection unit 21 that detects an object in an image captured by the image capture device 10 and generates object information, a resizing unit 23 that compares the object information with a predetermined threshold to determine whether the degree of adverse impact due to information loss when the image is reduced exceeds a standard, and a resizing unit 24 that does not reduce the image if it is determined that the degree of adverse impact due to information loss exceeds the standard, and an encoding unit 24 that encodes the image processed by the resizing unit 23. Furthermore, a restoration processing device 50 on the receiving side includes a decoding unit 51 that decodes data received from the transmitting side, and a restoration unit 52 that restores the image to its original resolution before reduction if the image decoded by the decoding unit 51 has been reduced. This configuration minimizes information loss in the image to a practically acceptable level, while applying image reduction and restoration when possible, thereby improving the efficiency of data transmission.

[0050] In the above explanation, a threshold value related to the area of ​​the bounding box is used as the predetermined threshold value, and if the area of ​​the smallest bounding box among one or more bounding boxes included in the object information that is the object detection result is smaller than the predetermined threshold value, the resizing unit 23 determines that the degree of adverse effect due to information loss exceeds the standard and does not reduce the image, but reduces the image otherwise.However, this method is an example, and the appropriateness of image reduction may also be determined using other methods as described below.

[0051] Next, some modifications of the image transmission system described above will be described, but the present invention is not limited to these.

[0052] (First Modification) In the above-described embodiment, an example was shown in which the area threshold θ was uniquely set in step S11. However, instead of this, a different threshold θ is set for each class of object to be detected. That is, the class IDs of classes to be detected by the object detection unit 22 are set to 1, 2, ..., n, and the threshold θ is set for each class. 1 , θ 2 , ..., θ n In steps S15 and S16, the smallest bounding box is obtained for each class of detected object, and it is evaluated whether the area of ​​each class is larger than a threshold value.

[0053] Threshold θ i When determining i (i = 1 to n), the threshold value can be set taking into consideration the apparent complexity of the class, the importance of the class, etc. For example, if it is necessary to capture a person's hands and feet more clearly, the threshold value for the person class can be set to a large value. Also, for objects that are not visually complex and are easy to distinguish, such as balls, the threshold value can be set to a small value.

[0054] (Second Modification) In the above-described embodiment, an example was shown in which the determination of whether to downsize was made based on the area in step S16, but instead, the determination may be made based on the magnitude relationship between the score obtained by object detection rather than the area, and a threshold. That is, if the smallest score among one or more scores included in the object information is smaller than the threshold, it is determined that the degree of adverse effect due to information loss exceeds the threshold, and the image is not downsized, but if not, the image is downsized.

[0055] The score used here is a score that indicates the degree of certainty regarding an object detected from an image, and for example, the certainty scores (scores that indicate the degree of certainty of the object class) shown in FIG. 8(b) and FIG. 9(b) can be used. In addition, some object detection methods internally use a score called objectness, and this can also be used. Objectness is a score that indicates the degree to which the target is determined to be an object, regardless of its class (scores that indicate the degree of certainty as an object).

[0056] (Third Modification) When reducing the image in step S17, the reduction ratio can be arbitrarily determined. For example, a fixed ratio can be set, such as reducing the image so that the number of pixels in the vertical and horizontal directions is halved, or the smallest bounding box B among the areas acquired in step S15 can be set. min For example, the reduction ratio may be dynamically determined according to the area of ​​θ / (B min By setting the area of ​​B after reduction min It is possible to reduce the area of ​​the pixel to a constant value that does not fall below a threshold value.

[0057] (Fourth Modification) If the resolution of the original image is known in the restoration unit 52 of the restoration processing device 50, it can easily determine whether the image has been reduced, but if not, any determination method can be used. For example, by embedding resolution information of the original image as metadata in the encoded image data in the encoding unit 24 of the conversion processing device 20, the restoration processing device 50 can obtain the resolution of the original image by reading the metadata.

[0058] Although the embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. For example, the above-described embodiments and the first to fourth modifications can be applied individually, and of course, they can also be combined in any desired manner. The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and their modifications are included within the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents.

[0059] Furthermore, the present invention can be provided not only as devices such as those described above or as systems composed of these devices, but also as methods executed by these devices, programs for realizing the functions of these devices using a processor, and storage media for storing such programs in a computer-readable manner.

[0060] The present invention can be used in a conversion processing device that converts images captured by a photographing device for transmission, and in an image transmission system that uses the same.

[0061] 10: Image capture device, 20: Conversion processing device, 21: Image acquisition unit, 22: Object detection unit, 23: Resize unit, 24: Encode unit, 30: Communication device (transmitter), 40: Communication device (receiver), 50: Restoration processing device, 51: Decode unit, 52: Restoration unit, 53: Storage unit, 201, 501: Processor, 202, 502: Main memory, 503: Storage

Claims

1. A conversion processing device that converts images captured by a photographing device for transmission, comprising: an object detection unit that detects an object from the image and generates object information; a resizing unit that compares the object information with a predetermined threshold to determine whether the degree of adverse effect due to information loss when the image is reduced exceeds a standard, and does not reduce the image if it is determined that the degree of adverse effect due to information loss exceeds the standard, but reduces the image otherwise; and an encoding unit that encodes the image that has been processed by the resizing unit.

2. A conversion processing device according to claim 1, wherein the object detection unit detects objects of a plurality of different classes, and the threshold value is set for each class.

3. A conversion processing device according to claim 1, wherein the object information includes a bounding box that indicates the position and size of an object detected from the image using a rectangle, the predetermined threshold is a threshold related to the area of ​​the bounding box, and the resizing unit determines that the degree of adverse effect due to information loss exceeds a standard when the area of ​​the smallest bounding box among one or more bounding boxes included in the object information is smaller than the predetermined threshold and does not reduce the image, but reduces the image otherwise.

4. A conversion processing device according to claim 1, wherein the object information includes a score representing a degree of certainty regarding an object detected from the image, the predetermined threshold is a threshold regarding the score, and the resizing unit determines that the degree of adverse effect due to information loss exceeds a standard when the smallest score among one or more scores included in the object information is smaller than the predetermined threshold and does not reduce the image, but reduces the image otherwise.

5. A conversion processing device according to claim 1, wherein the resizing section determines a magnification for reducing the image in accordance with the object information.

6. A conversion processing device according to claim 1, wherein when an image is reduced by said resizing section, said encoding section embeds information about the resolution before reduction as metadata of the reduced image.

7. An image transmission system for transmitting images captured by a photographing device from a sending side to a receiving side, wherein the conversion processing device provided on the sending side comprises: an object detection unit that detects an object from the image and generates object information; a resizing unit that compares the object information with a predetermined threshold to determine whether the degree of adverse effect due to information loss when the image is reduced exceeds a standard, and does not reduce the image if it is determined that the degree of adverse effect due to information loss exceeds the standard, but reduces the image otherwise; and an encoding unit that encodes the image that has been processed by the resizing unit; and the restoration processing device provided on the receiving side comprises: a decoding unit that decodes data received from the sending side; and a restoration unit that, if the image decoded by the decoding unit has been reduced, restores the image to the resolution before reduction.

Images