Image compression device, image compression method, computer program, image compression system, and image processing system

Abstract

This image compression device comprises: a target region extraction unit for extracting a target region which includes an object of a prescribed size from an image; and an image compression unit for compressing the image on the basis of the result of extracting the target region.

Description

[Technical field]

[0001] The present disclosure relates to an image compression device, an image compression method, a computer program, an image compression system, and an image processing system. This application claims priority to Japanese Application No. 2020-167734, filed on October 2, 2020, and incorporates by reference all of the contents of said Japanese application. [Background technology]

[0002] With the recent progress of AI (artificial intelligence) technology, such as deep learning, image compression technology using AI has been researched (for example, see Non-Patent Document 1).

[0003] In the technology disclosed in Non-Patent Document 1, a convolutional neural network (CNN) trained by deep learning is used to calculate the saliency of each pixel in an image. Here, saliency is a measure of how noticeable a pixel is to human vision. Non-Patent Document 1 discloses a compression method in which the compression rate is lowered for pixels with higher saliency. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] A. Prakash, N. Moran, S. Garber, A. Dilillo and J. Storer, 'Semantic Perceptual Image Compression Using Deep Convolution Networks,' 2017 Data Compression Conference (DCC), Snowbird, UT, 2017, pp. 250-259, doi: 10.1109 / DCC.2017.56. Summary of the Invention

[0005] An image compression device according to one aspect of the present disclosure includes an object area extraction unit that extracts an object area from an image, the object area being an area that includes an object of a predetermined size, and an image compression unit that compresses the image based on the result of the object area extraction.

[0006] An image compression method according to another aspect of the present disclosure includes a step of extracting a target area from an image, the target area being an area containing an object of a predetermined size, and a step of compressing the image based on the result of the extraction of the target area.

[0007] A computer program according to another aspect of the present disclosure causes a computer to function as a target area extraction unit that extracts a target area from an image, the target area being an area that includes an object of a predetermined size, and an image compression unit that compresses the image based on the result of the target area extraction.

[0008] An image compression system according to another aspect of the present disclosure includes a camera mounted on a moving object and the above-described image compression device that compresses an image captured by the camera.

[0009] An image processing system according to another aspect of the present disclosure includes the above-described image compression device, and an image decompression device that acquires a compressed image from the image compression device and decompresses the acquired compressed image.

[0010] It goes without saying that the computer program can be distributed through a computer-readable non-transitory recording medium such as a CD-ROM (Compact Disc-Read Only Memory) or a communication network such as the Internet. The present disclosure can also be realized as a semiconductor integrated circuit that realizes a part or all of the image compression device. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an overall configuration of a driving assistance system according to a first embodiment of the present disclosure. [Diagram 2]FIG. 2 is a block diagram showing an example of a configuration of the in-vehicle system according to the first embodiment of the present disclosure. [Diagram 3] FIG. 3 is a block diagram illustrating a functional configuration of the processor according to the first embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram showing an example of an image acquired by the image acquisition unit from the camera. [Diagram 5] FIG. 5 is a diagram for explaining a method of extracting a target region by the target region extraction unit. [Figure 6] FIG. 6 is a diagram for explaining a method of extracting a target region by the target region extraction unit. [Figure 7] FIG. 7 is a flowchart showing a processing procedure of the in-vehicle system according to the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a flowchart showing details of the image compression process (step S3 in FIG. 7). [Figure 9A] FIG. 9A is a diagram showing an example of a matrix of DCT (Discrete Cosine Transform) coefficients that is a result of a discrete cosine transform. [Figure 9B] FIG. 9B is a diagram showing an example of DCT coefficients after the DCT coefficients shown in FIG. 9A are quantized using the first quantization table. [Figure 9C] FIG. 9C is a diagram showing an example of DCT coefficients after the DCT coefficients shown in FIG. 9A are quantized using the second quantization table. [Figure 10] FIG. 10 is a block diagram illustrating an example of a configuration of the server according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a flowchart illustrating a processing procedure of the server according to the first embodiment of the present disclosure. [Figure 12] FIG. 12 is a flowchart showing details of the image decompression process (step S23 in FIG. 11). [Figure 13] FIG. 13 is a diagram for explaining the object detection method according to the first embodiment. [Figure 14] FIG. 14 is a diagram for explaining an object detection method using a conventional method. [Figure 15] FIG. 15 is a diagram showing experimental results of the object detection method according to the first embodiment and the object detection method using a conventional method. [Figure 16] FIG. 16 is a block diagram showing a functional configuration of a processor included in the in-vehicle system according to the second embodiment of the present disclosure. [Figure 17] FIG. 17 is a diagram showing an example of a prediction target frame. [Figure 18] FIG. 18 is a flowchart showing a processing procedure of the in-vehicle system according to the second embodiment of the present disclosure. [Figure 19] FIG. 19 is a diagram showing an example of an object extracted from an input image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Problem that this disclosure aims to solve] Conventional image compression methods are based on the premise that when a compressed image is expanded, parts that stand out to the human eye are made to look clear, and objects that are not noticeable to the human eye end up being compressed at a high compression rate.

[0013] For this reason, when the decompressed image is input to an object recognition device designed to recognize a specific object from within an image, it becomes difficult to recognize objects that are not noticeable to human vision. For example, when a camera is mounted on a moving object such as a car, it is necessary to accurately recognize small cars that are far away. This is because the system can provide driving assistance at an early stage by recognizing distant cars.

[0014] The present disclosure has been made in consideration of the above circumstances, and aims to provide an image compression device, an image compression method, a computer program, an image compression system, and an image processing system that are capable of achieving image compression at a high compression rate and accurate object recognition from the decompressed image. [Effects of this disclosure]

[0015] According to the present disclosure, it is possible to realize image compression at a high compression rate and accurate object recognition from the decompressed image.

[0016] [Description of the embodiments of the present disclosure] First, an overview of the embodiments of the present disclosure will be listed and described. (1) An image compression device according to one embodiment of the present disclosure includes a target area extraction unit that extracts a target area from an image, the target area being an area that includes an object of a predetermined size, and an image compression unit that compresses the image based on the result of the target area extraction.

[0017] According to this configuration, by setting the compression ratio of the target area to a level that enables accurate recognition of objects of a specified size contained in the target area when a compressed image is decompressed and object recognition is performed, it is possible to achieve image compression at a high compression ratio and accurate object recognition from the decompressed image.

[0018] (2) Preferably, the image compression section compresses the image so that a compression rate in the target region in the image is lower than a compression rate in a region other than the target region in the image.

[0019] According to this configuration, the target region can be compressed at a lower compression rate than the region other than the target region. For example, by setting the predetermined size to a size that includes small objects, it is possible to achieve image compression at a high compression rate and accurate object recognition from the decompressed image.

[0020] (3) More preferably, the target area extraction unit further extracts a type of the object included in the target area, and the image compression unit further adds information on the type of the object to the compressed image.

[0021] According to this configuration, when a compressed image is decompressed and object recognition is performed, processing can be performed according to the type of object.

[0022] (4) The target region extraction unit may extract the target region, which is a region including an object of the predetermined size, the object being of a type corresponding to an intended use of the compressed image.

[0023] According to this configuration, the type of object to be processed can be changed depending on the intended use of the compressed image, thereby realizing object recognition according to the intended use.

[0024] (5) The predetermined size may vary depending on the type of the object.

[0025] According to this configuration, it is possible to extract a target region of an appropriate size according to the type of object. For example, by setting a predetermined size for a car that is larger than that for a human, it is possible to appropriately extract target regions that include both a car and a human.

[0026] (6) The image compression unit may compress the image at a compression rate according to the type of the object included in the target area.

[0027] According to this configuration, the compression ratio can be changed for each type of object, so that, for example, by compressing an object type for which recognition accuracy is more important at a lower compression ratio, it is possible to accurately recognize important objects from the decompressed image.

[0028] (7) Furthermore, the above-mentioned image compression device may further include a target area prediction unit that predicts the target area in a first image based on the target area extracted from a first image taken at a first time and a second image taken at a second time different from the first time, and the image compression unit may compress the second image based on the prediction result by the target area prediction unit.

[0029] According to this configuration, it is possible to omit the process of extracting the target region from the second image, thereby enabling high-speed image compression processing.

[0030] (8) Additionally, the target area prediction unit may predict movement of the target area based on the target area extracted from the first image and the second image, and predict the target area in the second image based on the predicted movement and the target area extracted from the first image.

[0031] According to this configuration, the target area in the second image can be predicted from the movement of the target area, thereby making it possible to accurately predict the target area in the second image.

[0032] (9) Furthermore, the camera for capturing the image may be mounted on a moving object.

[0033] According to this configuration, the compressed image can be used to assist safe driving of a mobile object.

[0034] (10) An image compression method according to another embodiment of the present disclosure includes the steps of extracting a target area from an image, the target area being an area containing an object of a predetermined size, and compressing the image based on the result of the extraction of the target area.

[0035] This configuration includes the characteristic processing steps of the above-mentioned image compression device, and therefore, according to this configuration, it is possible to achieve the same functions and effects as the above-mentioned image compression device.

[0036] (11) A computer program according to another embodiment of the present disclosure causes a computer to function as a target area extraction unit that extracts a target area from an image, the target area being an area that includes an object of a predetermined size, and an image compression unit that compresses the image based on the result of the target area extraction.

[0037] According to this configuration, the computer can function as the above-mentioned image compression device, and therefore, the same effects and advantages as those of the above-mentioned image compression device can be achieved.

[0038] (12) An image compression system according to another embodiment of the present disclosure includes a camera mounted on a moving object and the above-described image compression device that compresses an image captured by the camera.

[0039] According to this configuration, by setting the compression ratio of the target area to a level at which an object of a predetermined size contained in the target area can be accurately recognized when the compressed image is expanded and object recognition is performed, it is possible to realize image compression at a high compression ratio and accurate object recognition from the expanded image. In addition, the compressed image can be used to support safe driving of a moving object.

[0040] (13) An image processing system according to another embodiment of the present disclosure includes the above-described image compression device and an image decompression device that acquires a compressed image from the image compression device and decompresses the acquired compressed image.

[0041] According to this configuration, by setting the compression ratio of the target area to a level that enables accurate recognition of objects of a specified size contained in the target area when a compressed image is decompressed and object recognition is performed, it is possible to achieve image compression at a high compression ratio and accurate object recognition from the decompressed image.

[0042] [Details of the embodiment of the present disclosure] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, the arrangement and connection forms of the components, steps, and the order of steps shown in the following embodiments are examples and do not limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims are components that can be added arbitrarily. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.

[0043] In addition, the same components are denoted by the same reference numerals, and since their functions and names are also the same, their explanations will be omitted as appropriate.

[0044] <Embodiment 1> [Overall configuration of driving assistance system] FIG. 1 is a diagram showing an overall configuration of a driving assistance system according to a first embodiment of the present disclosure.

[0045] Referring to FIG. 1, the driving assistance system 1 includes a plurality of vehicles 2 traveling on a road where wireless communication is possible, one or more base stations 6 that wirelessly communicate with the vehicles 2, and a server 4 that communicates with the base stations 6 via a network 5 such as the Internet in a wired or wireless manner.

[0046] The base station 6 includes a macrocell base station, a microcell base station, a picocell base station, and the like.

[0047] The vehicle 2 includes not only ordinary passenger cars (automobiles) but also public vehicles such as route buses and emergency vehicles. Furthermore, the vehicle 2 may be not only a four-wheeled vehicle but also a two-wheeled vehicle (motorcycle, motorcycle).

[0048] Each vehicle 2 is equipped with an on-board system 3 including a camera, as described below, which compresses image data (hereinafter simply referred to as "images") obtained by photographing the surroundings of the vehicle 2 with the camera, and transmits the compressed images to a server 4 via a network 5.

[0049] The server 4 receives compressed images from each vehicle 2 via the network 5 and decompresses the received compressed images. The server 4 performs a predetermined image processing on the decompressed images. For example, the server 4 executes a recognition process to recognize the vehicles 2, people, traffic signals, and road signs from the images, and creates a dynamic map that reflects the recognition results on the map data. The server 4 transmits the created dynamic map to each vehicle 2.

[0050] Each vehicle 2 receives the dynamic map from the server 4, and performs driving assistance processing and the like for the vehicle 2 based on the received dynamic map.

[0051] [Configuration of in-vehicle system 3] FIG. 2 is a block diagram showing an example of the configuration of the in-vehicle system 3 according to the first embodiment of the present disclosure.

[0052] As shown in FIG. 2, the in-vehicle system 3 of the vehicle 2 includes a camera 31, a communication unit 32, and a control unit (ECU: Electronic Control Unit) 33.

[0053] The camera 31 is mounted on the vehicle 2 and is made up of an image sensor that captures images of the surroundings of the vehicle 2 (particularly, the area ahead of the vehicle 2). The camera 31 is monocular. However, the camera 31 may be compound-eyed. The images are made up of a plurality of images in a time series.

[0054] The communication unit 32 is, for example, a wireless communication device capable of communication processing compatible with 5G (fifth generation mobile communication system). The communication unit 32 may be a wireless communication device already installed in the vehicle 2, or may be a portable terminal brought into the vehicle 2 by a passenger.

[0055] The passenger's mobile terminal is connected to an in-vehicle LAN (Local Area Network) of the vehicle 2, and temporarily becomes an in-vehicle wireless communication device.

[0056] The control unit 33 is composed of a computer device that controls the on-board devices mounted on the vehicle 2, including the camera 31 and the communication unit 32 of the vehicle 2. The on-board devices include, for example, a GPS receiver, a gyro sensor, and the like. The control unit 33 obtains the vehicle position of the vehicle itself based on the GPS signal received by the GPS receiver. The control unit 33 also determines the direction of the vehicle 2 based on the detection result of the gyro sensor.

[0057] The control unit 33 includes a processor 34 and a memory 35 . The processor 34 is an arithmetic processing device such as a microcomputer that executes a computer program stored in the memory 35 .

[0058] The memory 35 is composed of a volatile memory element such as a static RAM (SRAM) or a dynamic RAM (DRAM), a non-volatile memory element such as a flash memory or an electrically erasable programmable read only memory (EEPROM), or a magnetic storage device such as a hard disk, etc. The memory 35 stores computer programs executed by the control unit 33, data generated when the control unit 33 executes the computer programs, etc.

[0059] [Functional configuration of processor 34] FIG. 3 is a block diagram showing a functional configuration of the processor 34 according to the first embodiment of the present disclosure.

[0060] Referring to FIG. 3, the processor 34 includes an image acquisition unit 36, a target area extraction unit 37, and an image compression unit 38 as functional processing units realized by executing a computer program stored in the memory 35.

[0061] The image acquisition unit 36 ​​sequentially acquires, in time series, images of the area ahead of the vehicle 2 captured by the camera 31. The image acquisition unit 36 ​​sequentially outputs the acquired images to the target area extraction unit 37 and the image compression unit 38.

[0062] FIG. 4 is a diagram showing an example of an image (hereinafter referred to as an "input image") acquired by image acquisition section 36 from camera 31. As shown in FIG.

[0063] For example, the input image 50 includes a car 52 and a motorcycle 53 traveling on a road 51, and a person 55 walking on a crosswalk 54 installed on the road 51. The input image 50 also includes a road sign 56.

[0064] 3 again, target region extraction section 37 acquires input image 50 from image acquisition section 36, and extracts a target region that is a region including an object of a predetermined size from input image 50. A method for extracting the target region will be specifically described below.

[0065] FIG. 5 and FIG. 6 are diagrams for explaining a method of extracting a target region by the target region extraction unit 37.

[0066] Referring to FIG. 5, the target region extraction unit 37 divides the input image 50 into a plurality of blocks 60. In FIG. 5, as an example, an example in which the input image 50 is divided into 64 (= 8 × 8) blocks 60 is shown. The size of the block 60 is determined in advance, and all of them may have the same size, or some or all of them may have different sizes.

[0067] The target region extraction unit 37 determines whether an object of a predetermined size is included in the block image by inputting the image of each block (hereinafter referred to as "block image") into the learning model. Here, the object of a predetermined size is, for example, an object that satisfies the following formula (1). However, sqrt(x) is the square root of x, and a and b are constants (where a < b). a < sqrt (the number of pixels included in the circumscribed rectangle of the object) < b … (Formula 1)

[0068] In Embodiment 1, by setting a and b to small values, it is determined whether a small object is included in the block 60.

[0069] Note that the learning model is, for example, a CNN (Convolution Neural Network), an RNN (Recurrent Neural Network), an AutoEncoder, or the like. It is assumed that each parameter of the learning model is determined by a machine learning method such as deep learning using a block image including an object that satisfies Formula 1 and the type of the object (hereinafter referred to as "object type") as teacher data.

[0070] That is, the target area extraction unit 37 inputs an unknown block image into the learning model, and calculates the degree of certainty that the block image contains an object that satisfies formula 1 for each object type. The target area extraction unit 37 extracts a block with a certainty degree equal to or greater than a predetermined threshold for each object type as a target area, and extracts the object type at the time of extraction as the object type of the object contained in the target area. The target area extraction unit 37 outputs information on the extracted target area and object type to the image compression unit 38. Note that the target area information includes, for example, the upper left corner coordinates and the lower right corner coordinates of the target area. However, the method of expressing the target area is not limited to this. For example, the target area information may include the upper left corner coordinates of the target area and the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the target area, or may include an identifier indicating the target area.

[0071] Here, the object type indicates the type of object. In the first embodiment, the image is used for driving assistance of the vehicle 2. For this reason, the object type includes vehicles including two-wheeled vehicles and four-wheeled vehicles, humans, road signs, and traffic signals. Note that the object type is not limited to these. For example, bicycles may be included as a type separate from vehicles.

[0072] The object type may be different depending on the use of the image. For example, if the camera 31 is installed on a forklift that travels in a factory and the image is used for monitoring the factory, the object types include vehicles, people, and road signs, but do not necessarily include traffic lights. This is because some factories do not have traffic lights.

[0073] Furthermore, when the image is used for the purpose of delivering a package, the delivery support process may be performed based on landmarks. For this reason, for example, the object types may include landmarks such as buildings and signs.

[0074] It is assumed that road sign 56, person 55, and motorcycle 53 satisfy formula 1. Therefore, referring to Fig. 6, target area extraction unit 37 extracts target area 61 and road sign, target area 62 and person, and target area 63 and vehicle as pairs of target areas and object types.

[0075] It is assumed that the automobile 52 does not satisfy Expression 1. For this reason, the target region extraction unit 37 does not extract the automobile 52 as a target region. A block that is not extracted as a target region is called a non-target region 65.

[0076] Referring again to FIG. 3, the image compression unit 38 acquires the input image 50 from the image acquisition unit 36, and acquires information on the target area and object type from the target area extraction unit 37. The image compression unit 38 compresses the input image 50 for each block. At that time, the image compression unit 38 compresses the target area and the non-target area at different compression rates. Specifically, the image compression unit 38 compresses the input image 50 so that the compression rate in the target area is lower than the compression rate in the non-target area. Here, the compression rate is the amount of data of the block before compression divided by the amount of data of the block after compression. Therefore, the amount of data after compression of the non-target area is smaller than the amount of data after compression of the target area. As a result, the target area can be compressed at a high compression rate when viewed as the entire image while maintaining its identity with the input image 50. Details of the compression process by the image compression unit 38 will be described later.

[0077] The image compression unit 38 adds information on the target area and object type to the compressed input image 50 and transmits the image to the server 4 via the communication unit 32 .

[0078] The processor 34 may receive a dynamic map from the server 4 and perform driving assistance processing for the vehicle 2 or the like based on the received dynamic map.

[0079] [Processing flow of in-vehicle system 3] FIG. 7 is a flowchart showing a processing procedure of the in-vehicle system 3 according to the first embodiment of the present disclosure.

[0080] The image acquisition unit 36 ​​acquires an image from the camera 31 (step S1). The target area extraction unit 37 extracts a target area and an object type from the input image 50 (step S2).

[0081] The image compression unit 38 compresses the input image 50 based on the input image 50 and the target region and object type extracted by the target region extraction unit 37 (step S3).

[0082] Fig. 8 is a flowchart showing details of the image compression process (step S3 in Fig. 7). The image compression process shown in Fig. 8 is an application of JPEG (Joint Photographic Experts Group) compression.

[0083] 8, the image compression unit 38 converts the color system of the input image 50 (step S11). That is, each pixel of the input image 50 includes an R signal, a G signal, and a B signal of the RGB color system. The image compression unit 38 converts the R signal, the G signal, and the B signal of the RGB color system into a Y signal, a Cb signal, and a Cr signal of the YCbCr color system for each pixel (step S11).

[0084] The image compression unit 38 repeatedly executes the processes of steps S12 to S16 described below for each block 60 included in the input image 50 (loop A).

[0085] That is, the image compression unit 38 performs a discrete cosine transform on the block 60 to be processed (step S12). Fig. 9A is a diagram showing an example of a matrix of DCT coefficients that is a result of the discrete cosine transform. The matrix has 8 rows x 8 columns of DCT coefficients as elements, and the DCT coefficients indicate the frequency components in the block 60. The upper left of the matrix indicates lower frequency components, and the lower right indicates higher frequency components.

[0086] The image compression unit 38 determines whether the block 60 to be processed is a target area or a non-target area based on the information acquired from the target area extraction unit 37 (step S13).

[0087] If the block 60 to be processed is a target region (YES in step S13), the image compression unit 38 quantizes the DCT coefficients using the first quantization table (step S14). On the other hand, if the block 60 to be processed is a non-target region (NO in step S13), the image compression unit 38 quantizes the DCT coefficients using the second quantization table (step S15). That is, the image compression unit 38 quantizes each DCT coefficient shown in FIG. 9A by the quantization coefficient at the corresponding position in the 8-row x 8-column quantization table.

[0088] Here, it is assumed that the first quantization table and the second quantization table are determined such that the number of levels after quantization using the first quantization table is greater than the number of levels after quantization using the second quantization table. In other words, when the first quantization table and the second quantization table are compared at the same row and column position, the quantization coefficient of the first quantization table is smaller than the quantization coefficient of the second quantization table.

[0089] Fig. 9B is a diagram showing an example of DCT coefficients after the DCT coefficients shown in Fig. 9A are quantized using a first quantization table, and Fig. 9C is a diagram showing an example of DCT coefficients after the DCT coefficients shown in Fig. 9A are quantized using a second quantization table.

[0090] For example, the DCT coefficients after quantization using the first quantization table shown in FIG. 9B have 32 levels from 0 to 31, and the DCT coefficients after quantization using the second quantization table shown in FIG. 9C have 10 levels from 0 to 9.

[0091] Referring again to FIG. 8, the image compression unit 38 performs run-length compression on the quantized DCT coefficients, and Huffman-encodes the run-lengths (step S16).

[0092] Referring again to FIG. 7, the image compression unit 38 adds information on the object type and the object region extracted by the object region extraction unit 37 to the compressed input image 50 (step S4).

[0093] The image compression unit 38 transmits the compressed input image 50 to which the target area information and object type information have been added in step S4 to the server 4 via the communication unit 32 (step S5).

[0094] [Server 4 configuration] FIG. 10 is a block diagram showing an example of the configuration of the server 4 according to the first embodiment of the present disclosure.

[0095] 10, the server 4 includes a communication unit 41 and a processor 42. However, the server 4 is a general computer including a CPU, a ROM, a RAM, etc., and FIG. 10 only shows some of these.

[0096] The communication unit 41 is a communication module that connects the server 4 to the network 5. The communication unit 41 receives the compressed image from the vehicle 2 via the server 4.

[0097] The processor 42 is composed of a CPU etc., and is equipped with a compressed image acquisition unit 43, an information extraction unit 44, an image decompression unit 45, and an image processing unit 46 as functional processing units realized by executing a computer program stored in a memory such as a ROM or RAM.

[0098] The compressed image acquisition unit 43 acquires the compressed image from the vehicle 2 via the communication unit 41. The compressed image acquisition unit 43 outputs the acquired compressed image to the information extraction unit 44 and the image decompression unit 45.

[0099] The information extraction unit 44 acquires the compressed image from the compressed image acquisition unit 43. The information extraction unit 44 extracts the target area information and object type information added to the compressed image from the compressed image. The information extraction unit 44 outputs the extracted information to the image decompression unit 45 and the image processing unit 46.

[0100] The image decompression unit 45 acquires the compressed image from the compressed image acquisition unit 43, and acquires the target area information from the information extraction unit 44. The image decompression unit 45 decompresses the compressed image based on the target area information. That is, the image decompression unit 45 decompresses the target area using a decompression method corresponding to the compression method of the target area, and decompresses the non-target area using a decompression method corresponding to the compression method of the non-target area. The method of decompressing the compressed image by the image decompression unit 45 will be described later. The image decompression unit 45 outputs the decompressed image to the image processing unit 46.

[0101] The image processing unit 46 obtains the target region information and object type information from the information extraction unit 44 , and obtains the decompressed image from the image decompression unit 45 .

[0102] The image processing unit 46 performs a predetermined image processing on the expanded image based on the target area information and the object type information. As an example, the image processing unit 46 performs a recognition process on the target area using the object type as a clue. For example, when the object type is a road sign, the image processing unit 46 recognizes the road sign by performing a pattern matching process using pattern images of various road signs. This allows the recognition process to be performed efficiently and accurately.

[0103] The image processing unit 46 may create a dynamic map in which the recognition result is reflected on the map data, and transmit the dynamic map to each vehicle 2 via the communication unit 41.

[0104] [Server 4 processing flow] FIG. 11 is a flowchart showing a processing procedure of the server 4 according to the first embodiment of the present disclosure.

[0105] The compressed image acquisition unit 43 acquires the compressed image from the vehicle 2 via the communication unit 41 (step S21).

[0106] The information extraction unit 44 extracts the added target area information and object type information from the compressed image (step S22).

[0107] The image decompression unit 45 decompresses the compressed image based on the target area information (step S23).

[0108] Fig. 12 is a flow chart showing details of the image decompression process (step S23 in Fig. 11). The image decompression process shown in Fig. 12 is an application of JPEG decompression.

[0109] 12, the image decompression unit 45 repeatedly executes the processes of steps S31 to S35 described below for each block 60 included in the compressed image (loop B). Note that the blocks 60 included in the compressed image are the same as the blocks 60 included in the input image 50.

[0110] The image decompression unit 45 calculates a run length by Huffman inverse coding the data corresponding to the processing target block 60. The image decompression unit 45 also calculates quantized DCT coefficients by decompressing the calculated run length (step S31).

[0111] The image decompression unit 45 determines whether the block 60 to be processed is a target area based on the target area information acquired from the information extraction unit 44 (step S32).

[0112] If the block 60 to be processed is a target region (YES in step S32), the image decompression unit 45 calculates DCT coefficients by inverse quantizing the quantized DCT coefficients using the first quantization table (step S33). On the other hand, if the block 60 to be processed is a non-target region (NO in step S32), the image decompression unit 45 calculates DCT coefficients by inverse quantizing the quantized DCT coefficients using the second quantization table (step S34). Here, the first quantization table and the second quantization table are the same as the first quantization table and the second quantization table used by the image compression unit 38 of the in-vehicle system 3 to quantize the DCT coefficients.

[0113] For example, image decompression unit 45 performs inverse quantization by multiplying each compressed DCT coefficient shown in Fig. 9B by a quantization coefficient at a corresponding position in a first quantization table of 8 rows x 8 columns. Similarly, image decompression unit 45 performs inverse quantization by multiplying each compressed DCT coefficient shown in Fig. 9C by a quantization coefficient at a corresponding position in a second quantization table of 8 rows x 8 columns.

[0114] The image decompression unit 45 performs a discrete cosine transform on the dequantized 8-row by 8-column DCT coefficients to calculate the Y signal, Cb signal, and Cr signal for each pixel (S35).

[0115] After the processing from step S31 to step S35 is completed for all blocks 60 in the compressed image (loop B), the image expansion unit 45 converts the color system in the image (step S36). That is, each pixel in the image includes a Y signal, a Cb signal, and a Cr signal in the YCbCr color system. The image expansion unit 45 converts the Y signal, the Cb signal, and the Cr signal in the YCbCr color system into an R signal, a G signal, and a B signal in the RGB color system for each pixel (step S36).

[0116] 11 again, the image decompression unit 45 outputs the decompressed image to the image processing unit 46. The image processing unit 46 performs a predetermined image processing on the decompressed image based on the information acquired from the information extraction unit 44 (step S24). For example, the image processing unit 46 executes a recognition process for recognizing the vehicle 2, a person, a traffic signal, and a road sign from the image, and creates a dynamic map in which the recognition result is reflected on the map data.

[0117] [Comparison results] The following describes the results of a comparison between the object detection method according to the first embodiment and an object detection method using a conventional technique.

[0118] 13 is a diagram for explaining the object detection method according to the first embodiment. That is, the target area extraction unit 37 extracts a target area from an input image of aMB (Mega Byte) (step ST1). The image compression unit 38 performs JPEG compression at a low compression rate on the target area (step ST2). This compression method is the same as that described above. Note that the data amount of the target area after JPEG compression at the low compression rate is bMB.

[0119] The image decompression unit 45 performs JPEG decompression on the data of the target area compressed in step ST2 (step ST3). This decompression method is the same as that described above.

[0120] The image processing unit 46 detects a small object (i.e., an object having a size shown in Equation 1) from the target area after JPEG decompression in step ST3 (step ST4). Note that the machine learning model YOLOv3 (You Only Look Once v3) is used for object detection.

[0121] On the other hand, the target area extraction unit 37 performs JPEG compression with a higher compression rate than the JPEG compression of the target area on the entire input image (step ST5). This compression method is the same as the compression method for the non-target area described above. Note that the data amount of the image after JPEG compression with the high compression rate is cMB.

[0122] The image compressed in step ST5 is subjected to JPEG decompression by the image decompression unit 45 (step ST6). This decompression method is the same as the decompression method for the non-target region described above.

[0123] The image processing unit 46 detects a large object (that is, an object larger than the size shown in Equation 1) from the image after JPEG decompression in step ST6 (step ST7). Note that the machine learning model YOLOv3 is used for object detection.

[0124] The image processing unit 46 integrates the object detection result in step ST4 and the object detection result in step ST7. That is, when an object is detected in the same position in both step ST4 and step ST7, the image processing unit 46 selects the object with the highest certainty of object detection output by YOLOv3 as the detection result.

[0125] The compression rate of the input image obtained by performing compression in steps ST2 and ST5 is calculated by the following formula 2. Compression ratio = a / (b+c) ... (Equation 2)

[0126] FIG. 14 is a diagram for explaining an object detection method using a conventional method. That is, the image compression unit 38 performs normal JPEG compression on the entire input image (step ST11). Note that the data amount of the image after JPEG compression is dMB.

[0127] The image decompression unit 45 performs JPEG decompression on the image compressed in step ST11 (step ST12).

[0128] The image processing unit 46 detects an object from the image after JPEG decompression in step ST12 (ST13). The detection target object includes both the above-mentioned small object and large object. In addition, the machine learning model YOLOv3 is used for object detection.

[0129] The compression rate of the input image obtained by performing compression in step ST11 is calculated by the following formula 3. Compression ratio = a / d ... (Equation 3)

[0130] Fig. 15 is a diagram showing experimental results of the object detection method according to embodiment 1 and the object detection method using a conventional method. The horizontal axis of the graph shown in Fig. 15 indicates the compression ratio, and the vertical axis indicates the average recall ratio. The compression ratio is a value calculated using Equation 2 or Equation 3. The recall ratio indicates the ratio (percentage) between the number of objects that could be accurately detected from one image and the number of actual objects contained in the image. The average recall ratio indicates the average value of the recall ratios of multiple images.

[0131] As can be seen from the graph in Fig. 15, in the object detection method using the conventional method, the average recall rate drops sharply as the compression rate increases. In contrast, in the object detection method according to embodiment 1, the average recall rate drops only gradually even if the compression rate is increased. Also, at approximately the same compression rate (compression rate of about 150 times), the object detection method according to embodiment 1 has a higher average recall rate than the object detection method using the conventional method.

[0132] [Effects of the First Embodiment] As described above, the in-vehicle system 3 includes the target area extraction unit 37 that extracts a target area, which is an area including an object of a predetermined size, from the image captured by the camera 31, and the image compression unit 38 that compresses the image based on the target area extraction result. As a result, by setting the compression ratio of the target area to a level at which an object of a predetermined size included in the target area can be accurately recognized when the compressed image is expanded and object recognition is performed, it is possible to realize image compression at a high compression ratio and accurate object recognition from the expanded image.

[0133] In addition, the image compression unit 38 compresses the image so that the compression rate in the target area in the image is lower than the compression rate in the non-target area. Therefore, the target area can be compressed at a lower compression rate than the non-target area. For example, by setting the predetermined size to a size that includes small objects, it is possible to achieve image compression at a high compression rate and accurate object recognition from the decompressed image.

[0134] Furthermore, the target area extraction unit 37 further extracts the type of object included in the target area, and the image compression unit 38 further adds information on the type of object to the compressed image. Therefore, when the compressed image is expanded for object recognition, processing according to the type of object can be performed.

[0135] In addition, the target area extraction unit 37 extracts a target area that is an area including an object of a predetermined size, the object being of a type corresponding to the intended use of the compressed image. Therefore, the type of object to be processed can be changed for each intended use of the compressed image. This makes it possible to realize object recognition according to the intended use.

[0136] The camera 31 is mounted on the vehicle 2. Therefore, the compressed image can be utilized for supporting safe driving of the vehicle 2.

[0137] <Embodiment 2> In the first embodiment, the target area extraction unit 37 of the in-vehicle system 3 extracts a target area from each of the time-series images acquired from the camera 31. In the second embodiment, the target area is extracted from some of the time-series images, and the target area is predicted for the remaining images, which is different from the first embodiment.

[0138] The configuration of the driving assistance system 1 according to the second embodiment is similar to that of the first embodiment. However, the configuration of the in-vehicle system 3 is partially different from that of the first embodiment.

[0139] FIG. 16 is a block diagram showing a functional configuration of the processor 34 included in the in-vehicle system 3 according to the second embodiment of the present disclosure.

[0140] Referring to FIG. 16, the processor 34 includes an image acquisition unit 36, a target area extraction unit 37, an image compression unit 38, and a target area prediction unit 39 as functional processing units realized by executing a computer program stored in the memory 35.

[0141] The configuration of the image acquisition unit 36 ​​is the same as that of the embodiment 1. However, the image acquisition unit 36 ​​further outputs the input image to a target region prediction unit 39.

[0142] The configuration of the target region extraction unit 37 is the same as in the first embodiment. However, the target region extraction unit 37 extracts the target region from the extraction target frames among the time-series input images (frames), and does not extract the target region from the other frames. It is assumed that the extraction target frames are determined in advance. For example, odd-numbered frames among the time-series frames are determined as extraction target frames, and even-numbered frames are not determined as extraction target frames. It is noted that the method of determining the extraction target frames is not limited to this. For example, the extraction target frames may be selected every third frame. The target region extraction unit 37 outputs the target region information to the target region prediction unit 39 .

[0143] The target region prediction unit 39 acquires a frame other than the extraction target frame (hereinafter referred to as a “prediction target frame”) from the image acquisition unit 36. In addition, the target region prediction unit 39 acquires target region information from the target region extraction unit 37.

[0144] The target area prediction unit 39 predicts the target area in the second image based on the target area extracted from the first image captured by the camera 31 at the first time and the second image captured by the camera 31 at a second time different from the first time. For example, the first time is the capture time of an odd-numbered frame, and the second time is the capture time of an even-numbered frame. In other words, the target area prediction unit 39 predicts the target area in the prediction target frame based on the target area extracted from the extraction target frame and the prediction target frame.

[0145] Specifically, the target region prediction section 39 predicts the movement of the target region based on the target region extracted from the extraction target frame and the prediction target frame.

[0146] For example, when the input image 50 shown in FIG. 6 is the extraction target frame, the target region extraction unit 37 extracts the target region 61, the target region 62, and the target region 63. Here, FIG. 17 is a diagram showing an example of a prediction target frame. The input image 50 shown in FIG. 17 shows an example of a prediction target frame, and is a frame taken at a later time (for example, one frame later) than the extraction target frame shown in FIG. 6. The person 55 shown in FIG. 6 moves leftward in the input image 50, and the motorcycle 53 and the target region 63 move rightward and downward in the input image 50. The road sign 56 is not moving. It is assumed that the camera 31 is stopped. However, the camera 31 may be moving.

[0147] The target area prediction unit 39 calculates the motion vectors of the target area 61, the target area 62, and the target area 63 by performing a pattern matching process on the input image 50 shown in Fig. 17 using each of the target area 61, the target area 62, and the target area 63 shown in Fig. 6 as a template image. For example, when the centers of the target area 61, the target area 62, and the target area 63 are set as the starting points of the motion vectors, the end points of the motion vectors of the target area 61 and the target area 62 are set within the target area 61 and the target area 62, respectively. On the other hand, the end point of the motion vector of the target area 63 is set within the block one below.

[0148] The target area prediction unit 39 predicts the target area in the prediction target frame based on the target area and the calculated motion vector of the target area. For example, the target area prediction unit 39 predicts the target area 61 and the target area 62 as the target area because the end points of the motion vector for the target area 61 and the target area 62 are within the target area 61 and the target area 62, respectively. On the other hand, the target area prediction unit 39 predicts the target area 64 obtained by moving the target area 63 to the block below as the target area because the end point of the motion vector for the target area 63 is within the block below.

[0149] Although the target area prediction unit 39 performs pattern matching on a target area basis, the present invention is not limited to this. For example, the target area prediction unit 39 may extract an object such as a motorcycle 53, a person 55, or a road sign 56 from the target area, and calculate a motion vector by performing pattern matching processing using the image of the object as a template image. The target area prediction unit 39 may also determine that the block to which the end point of the motion vector belongs is the target area. The target region prediction unit 39 outputs information on the predicted target region to the image compression unit .

[0150] The image compression section 38 obtains target region information for the extraction target frame from the target region extraction section 37, and obtains target region information for the prediction target frame from the target region prediction section 39.

[0151] FIG. 18 is a flowchart showing a processing procedure of the in-vehicle system 3 according to the second embodiment of the present disclosure.

[0152] The image acquisition unit 36 ​​acquires an image from the camera 31 (step S1). The image acquisition unit 36 ​​determines whether or not the acquired image is a frame to be extracted (step S41).

[0153] If the acquired image is an extraction target frame (YES in step S41), the image acquisition unit 36 ​​outputs the extraction target frame to the target area extraction unit 37, and the target area extraction unit 37 extracts a target area and an object type from the extraction target frame (step S2).

[0154] If the acquired image is a prediction target frame (NO in step S41), the image acquisition unit 36 ​​outputs the prediction target frame to the target area prediction unit 39, and the target area prediction unit 39 calculates a motion vector from the target area extracted by the target area extraction unit 37 and the prediction target frame (step S42).

[0155] The target area prediction unit 39 predicts the target area in the prediction target frame based on the target area of ​​the extraction target frame extracted by the target area extraction unit 37 and the calculated motion vector. The target area prediction unit 39 also predicts the type of object corresponding to the target area of ​​the extraction target frame used for prediction as the type of object included in the predicted target area (step S43).

[0156] The image compression unit 38 compresses the extraction target frame based on the target area and object type extracted by the target area extraction unit 37, and compresses the prediction target frame based on the target area and object type predicted by the target area prediction unit 39 (step S3). Details of the image compression method are the same as those in the first embodiment.

[0157] The image compression unit 38 adds information on the target area and object type extracted by the target area extraction unit 37 to the compressed extraction target frame, and adds information on the target area and object type predicted by the target area prediction unit 39 to the compressed prediction target frame (step S4).

[0158] The image compression unit 38 transmits the compressed input image 50 to which the target area information and object type information have been added in step S4 to the server 4 via the communication unit 32 (step S5).

[0159] As described above, the in-vehicle system 3 further includes a target area prediction unit 39 that predicts a target area in a prediction target frame based on a target area extracted from a first image (extraction target frame) captured at a first time and a second image (prediction target frame) captured at a second time different from the first time. Furthermore, the image compression unit 38 compresses the prediction target frame based on the prediction result by the target area prediction unit 39. This makes it possible to omit the process of extracting the target area from the prediction target frame. This allows the image compression process to be performed at high speed.

[0160] Specifically, the target area prediction unit 39 predicts the movement of the target area based on the target area extracted from the extraction target frame and the prediction target frame, and predicts the target area in the prediction target frame based on the predicted movement and the target area extracted from the extraction target frame. In this way, the target area in the prediction target frame can be predicted from the movement of the target area. This makes it possible to accurately predict the target area in the prediction target frame.

[0161] <Variation 1> In the first and second embodiments, a block including an object of a predetermined size is extracted as the target region, but the method of extracting the target region is not limited to this.

[0162] For example, the target region extraction unit 37 may determine whether or not an object of a predetermined size is included in the input image 50 by inputting the input image 50 directly into a learning model. Here, the object of the predetermined size is, for example, an object that satisfies Expression 1.

[0163] The learning model is, for example, CNN, RNN, AutoEncoder, etc. It is assumed that each parameter of the learning model is determined by a machine learning method such as deep learning using an image including an object that satisfies Formula 1 and the object type as training data.

[0164] FIG. 19 is a diagram showing an example of an object extracted from an input image. For example, the target region extraction unit 37 inputs the input image 50 shown in Fig. 4 to the learning model. With reference to Fig. 19, the learning model extracts a motorcycle 53, a person 55, and a road sign 56 as objects included in the input image 50 and satisfying Expression 1. The target region extraction unit 37 also acquires the object types of the motorcycle 53, the person 55, and the road sign 56, that is, the vehicle, the person, and the road sign, from the learning model.

[0165] The image compression unit 38 performs compression processing in the same manner as in embodiment 1, treating the area containing the object extracted by the target area extraction unit 37 (for example, the circumscribing rectangular area of ​​the object or a block containing the object) as the target area, and treating the other areas as non-target areas.

[0166] <Variation 2> In the first and second embodiments, the predetermined size shown in formula 1 is the same even if the object type is different, but the predetermined size may be different for each object type. For example, humans and road signs are smaller than vehicles. Therefore, the predetermined size for humans and road signs is set smaller than the predetermined size for vehicles.

[0167] This makes it possible to extract a target region of an appropriate size according to the type of object. For example, by making a size of a car larger than that of a person, it is possible to appropriately extract target regions that include both a car and a person.

[0168] <Modification 3> In the first and second embodiments, the target region is compressed at the same compression rate even if the object type is different. However, the compression rate may be changed for each object type. In this way, for example, the more important the object type for which recognition accuracy is important, the lower the compression rate is used to compress the object, making it possible to accurately recognize important objects from the decompressed image.

[0169] [Note] The image compression method described above is not limited to JPEG compression, and a compression method that allows the compression rate to be changed or two or more compression methods with different compression rates may be used. For example, block data in the target area may be lossily compressed using an algorithm called Visually Lossless Compression or Visually Reversible Compression, which has a low compression rate. Also, block data in the non-target area may be compressed according to a compression method called JPEG2000, which has a high compression rate.

[0170] In addition, for the non-target region, a downscaling process may be performed to reduce the size of the non-target region, or the number of bits representing the luminance value of each pixel in the non-target region may be reduced to reduce the gradation (color depth).In addition, a temporal thinning process may be performed on the non-target region (for example, a process of deleting the non-target region obtained from even-numbered frames in the time-series images).

[0171] Some or all of the components constituting each of the above devices may be composed of one or more semiconductor devices such as a system LSI.

[0172] The computer program may be distributed by recording it on a non-transitory computer-readable recording medium, such as a HDD, a CD-ROM, a semiconductor memory, etc. The computer program may be distributed by transmitting it via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc. Furthermore, each of the above devices may be realized by multiple computers or multiple processors.

[0173] Furthermore, some or all of the functions of each of the above devices may be provided by cloud computing, that is, some or all of the functions of each device may be realized by a cloud server. Furthermore, the image compression section 38 may apply the present disclosure to a part of the range of the image captured by the camera 31. Furthermore, at least some of the above-described embodiments and modifications may be combined in any manner.

[0174] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present disclosure is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0175] 1 Driving assistance system (image processing system) 2 Vehicles 3. In-vehicle system (image compression system) 4 Server 5. Network 6 base station 31 Camera 32 Communications Department 33 Control unit (ECU) 34 Processor (Image Compression Device) 35 Memory 36 Image acquisition section 37 Target region extraction section 38 Image Compression Section 39 Target Area Prediction Department 41 Communications Department 42 processors 43 Compressed image acquisition unit 44 Information extraction part 45 Image Decompression Unit 46 Image Processing Section 50 input images 51 Road 52 Automobiles 53 Motorcycle 54 Crosswalk 55 humans 56 Road signs 60 blocks 61 Target Area 62 Target Area 63 Target Area 64 Target Area 65 Non-target areas

Claims

1. a target region extraction unit that divides an image into a plurality of block images, and determines whether or not each block image includes an object of a predetermined size, the object being an object whose size falls within a predetermined range, thereby extracting the block image including the object of the predetermined size as a target region; an image compression unit that determines a compression rate for each block image based on a result of the extraction of the target region and then compresses the image.

2. The image compression device according to claim 1 , wherein the image compression section compresses the image such that a compression rate in the target region in the image is lower than a compression rate in a region other than the target region in the image.

3. The target area extraction unit further extracts a type of the object included in the target area, 3. The image compression device according to claim 1, wherein the image compression section further adds information on a type of the object to the compressed image.

4. 4. The image compression device according to claim 1, wherein the target area extraction unit extracts the target area, which is an area including an object of a type corresponding to an intended use of the compressed image and an object of the predetermined size.

5. The image compression device according to claim 1 , wherein the predetermined size varies depending on a type of the object.

6. The image compression device according to claim 1 , wherein the image compression section compresses the image at a compression rate according to a type of the object included in the target region.

7. a target area prediction unit that predicts the target area in a second image based on the target area extracted from a first image captured at a first time and a second image captured at a second time different from the first time, The image compression device according to claim 1 , wherein the image compression section compresses the second image based on a prediction result by the target region prediction section.

8. 8. The image compression device according to claim 7, wherein the target region prediction unit predicts a movement of the target region based on the target region extracted from the first image and the second image, and predicts the target region in the second image based on the predicted movement and the target region extracted from the first image.

9. 9. The image compression device according to claim 1, wherein the camera for capturing the image is mounted on a moving object.

10. Dividing an image into a plurality of block images, and determining whether or not each block image includes an object of a predetermined size, the object size of which is within a predetermined range, thereby extracting the block image including the object of the predetermined size as a target region; determining a compression rate for each block image based on the result of extraction of the target region, and then compressing the image.

11. Computer, a target region extraction unit that divides an image into a plurality of block images, and determines whether or not each block image includes an object of a predetermined size, the object being an object whose size falls within a predetermined range, thereby extracting the block image including the object of the predetermined size as a target region; A computer program for causing the computer to function as an image compression unit that determines a compression rate for each block image based on a result of the extraction of the target area and then compresses the image.

12. A camera mounted on a moving object; An image compression system comprising: an image compression device according to any one of claims 1 to 9, which compresses an image captured by the camera.

13. An image compression device according to any one of claims 1 to 9; an image decompression device that acquires a compressed image from the image compression device and decompresses the acquired compressed image;

Images