Image processing device and image processing system

Abstract

To provide an image processing device and image processing system that can efficiently detect objects. [Solution] An image processing device according to one embodiment of the present invention comprises: an acquisition unit that acquires a sensor image generated by a sensor device; a first division unit that divides the acquired sensor image into predetermined sizes; a second division unit that divides the sensor image so as to straddle the division positions of the sensor image divided by the first division unit; a detection unit that detects a target to be detected within the first divided image divided by the first division unit; a determination unit that determines whether the coordinate position of the target to be detected includes the edge of the first divided image; and, if it is determined that the edge is included, an extraction unit that extracts an auxiliary image including the target to be detected from the second divided image divided by the second division unit.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing system for detecting an object from an input image.

Background Art

[0002] A technique for detecting an object from an image using a machine learning model is widely known. For example, Patent Document 1 describes a system for detecting a vehicle or a license plate from a captured image using a pre-trained model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In object detection, it is required to efficiently detect an object.

[0005] In view of the above circumstances, an object of the present invention is to provide an image processing apparatus and an image processing system capable of efficiently detecting an object.

Means for Solving the Problems

[0006] An image processing apparatus according to an aspect of the present invention includes an acquisition unit, a first division unit, a second division unit, a detection unit, a determination unit, and an extraction unit. The acquisition unit acquires a sensor image generated by a sensor device. The first division unit divides the acquired sensor image into a predetermined size. The second division unit divides the sensor image divided by the first division unit so as to straddle the division position. The detection unit detects the target to be detected within the first divided image divided by the first division unit. The determination unit determines whether the coordinate position of the detected target includes the edge of the first divided image. If the extraction unit determines that the end portion is included, it extracts an auxiliary image containing the detection target from the second divided image divided by the second division unit.

[0007] The second division portion described above may be generated by dividing the first division image so as to span across the first division image before the determination unit determines that it includes the end portion.

[0008] The second division described above may be further divided to include the detection target after it has been determined by the determination unit to include the end portion.

[0009] If the determination unit determines that the end portion is included, it further determines the first divided image adjacent to the end portion, The extraction unit may extract a second segmented image that spans a first segmented image including the end portion and a first segmented image adjacent to the end portion.

[0010] The system may further include a reading unit that reads information about the detection target from the target image extracted by the extraction unit.

[0011] An image processing system according to one embodiment of the present invention comprises a sensor device and an image processing device. The above sensor device generates a sensor image of the target being detected. The image processing apparatus includes an acquisition unit that acquires the sensor image generated by the sensor device, a first division unit that divides the acquired sensor image into a predetermined size, a second division unit that divides the sensor image so as to straddle the division position of the sensor image divided by the first division unit, a detection unit that detects a detection target in the first division image divided by the first division unit, a determination unit that determines whether the coordinate position of the detection target includes an end portion of the first division image, and an extraction unit that extracts an auxiliary image including the detection target from the second division image divided by the second division unit when it is determined that the end portion is included.

Advantages of the Invention

[0012] According to the present invention, an object can be efficiently detected.

Brief Description of the Drawings

[0013] [Figure 1] It is a block diagram showing an image processing system according to an embodiment of the present invention. [Figure 2] It is a sensor image generated by a sensor device. [Figure 3] It is a divided sensor image, where (A) is an image divided into six parts and (B) is an image divided into ten parts. [Figure 4] It is a diagram for explaining the processing of the second division unit. [Figure 5] It is a flowchart showing object detection processing. [Figure 6] It is a block diagram showing an image processing system according to a modification. <0000s80>It is a diagram showing a division example by the first division unit according to a modification.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0015] (Image Processing System) [[ID=X]] FIG. 1 is a block diagram showing an image processing system 100 according to an embodiment of the present invention, FIG. 2 is a sensor image G generated by the sensor device 10. FIG. 3 is a divided sensor image G, (A) is an image divided into six parts, (B) is an image divided into ten parts, and FIG. 4 is a diagram for explaining the processing of the second dividing unit 33. The image processing system 100 of the present embodiment includes a sensor device 10, a machine learning device 20, and an image processing device 30. The image processing system 100 of the present embodiment is configured to detect an object to be detected from an image of the sensor device 10 that captures a road such as an intersection, and transmit the determination result to a traffic control center or the like that manages traffic signals installed at the intersection or on the road.

[0016] In the present embodiment, the object to be detected is, for example, the license plate of a vehicle traveling on a road. Vehicles include automobiles, motorcycles, light vehicles, etc. In addition to this, the detection target can be arbitrarily set according to the detection purpose.

[0017] (Sensor device) The sensor device 10 is, for example, a video camera that captures a shooting area at a predetermined frame rate (for example, 30 frames / second). The installation location of the sensor device 10 is not particularly limited and may be a traffic signal, a utility pole near the shooting area, or the rooftop of a building. The sensor device 10 acquires the captured (generated) sensor image (captured image) and outputs this sensor image to the image processing device 30. In the present embodiment, the sensor device 10 is a video camera, but of course it is not limited to this, and it may be a LiDAR (Light Detection and Ranging), an infrared sensor, or the like. Thereby, the detection accuracy can be maintained even when halation occurs due to night or vehicle lights.

[0018] (Machine learning device) The machine learning device 20 is a device that generates learning parameters by learning the learning images, as a preliminary step to detecting a target by applying inference using learning parameters to the detection images divided into predetermined sizes when the sensor image generated by the sensor device 10 is divided into predetermined sizes, and comprises a control unit 21, a storage unit 22, and a communication unit 23.

[0019] The machine learning device 20 is configured to enable machine learning using deep learning on training images, and is configured to output the training parameters generated by machine learning to the image processing device 30. The machine learning device 20 may be a general-purpose device such as a PC, or it may be a dedicated device. The machine learning device 20 may also include a display unit (e.g., a liquid crystal display) and an operation unit (e.g., a keyboard, mouse, touch sensor, etc.).

[0020] The control unit 21 performs various calculations based on the various programs stored in the memory unit 22 and comprehensively controls each part of the machine learning device 20. Typically, the control unit 21 performs machine learning processing.

[0021] In the machine learning process, the control unit 21 performs learning on the training image, which includes the segmented detection target (license plate in this embodiment) from the detection image, and generates learning parameters. These learning parameters (learning model) are transmitted to the image processing device 30 and stored in the storage unit 38 of the image processing device 30.

[0022] The memory unit 22 includes a non-volatile memory for storing various programs and data necessary for processing by the control unit 21, and a volatile memory used as a work area for the control unit.

[0023] The communication unit 23 is configured to communicate with the image processing device 30.

[0024] (Image processing device) The image processing device 30 is typically composed of a computer equipped with a CPU (Central Processing Unit), memory, etc. The image processing device 30 includes an acquisition unit 31, a first division unit 32, a second division unit 33, a detection unit 34, a determination unit 35, an extraction unit 36, a reading unit 37, a storage unit 38, and a communication unit 39. Part of the CPU processing may be processed by a GPU.

[0025] The acquisition unit 31 acquires the sensor image G captured by the camera 10 and stores it in the storage unit 38.

[0026] The first division section 32 divides the sensor image G shown in Figure 2 into predetermined sizes as shown in Figure 3(A). In this embodiment, the first division section 32 divides the sensor image G into six sections as shown in Figure 3(A). In this embodiment, the size of the sensor image G is 1920 × 1080. The size of the division by the first division section 32 is 640 × 640. In this embodiment, the sensor image G and the size of the division were as described above, but of course, they are not limited to these.

[0027] As shown in Figure 3(A), of the first divided images B1 to B6 divided by the first division section 32, the top and bottom (divided images B1, B4, divided images B2, B5, divided images B3, B6) partially overlap, but of course this is not limited to this, and the top and bottom do not necessarily overlap. In this embodiment, the left and right (divided images B1, B2, divided images B2, B3, divided images B4, B5, divided images B5, B6) do not overlap, but of course this is not limited to this, and they may partially overlap.

[0028] The second division unit 33 divides the sensor image G so as to straddle the division positions of the sensor images (first divided images B1 to B6) divided by the first division unit 32. As shown in Figure 3(B), the second division unit 33 divides the sensor image G into second divided images B7 to B10. Here, "straddle" means that, as shown in Figures 3(B) and 3(4), the sensor image G is divided so as to include, for example, the division positions B11 and B21 of the divided images B1 and B2. In other words, the sensor image G is divided so that the second divided image B7 includes at least a part of each of the first divided images B1 and B2. In this embodiment, since there is some overlap vertically, only the left and right (non-overlapping) second divided images B7 to B10 were generated. However, if the divided images do not overlap vertically, the second divided images may be generated vertically (divided images B1, B4, divided images B2, B5, divided images B3, B6).

[0029] The detection unit 34 detects objects that have been pre-set as detection targets from the divided images (input images) divided by the first division unit 32 and the second division unit 33. In this embodiment, the detection unit 34 includes a deep learning model and uses the learning model stored in the memory unit 38 to determine whether or not a detection target exists in the input image. If a detection target exists, it detects its type (vehicle C, pedestrian, etc.) and location for each individual detection target. In this embodiment, the detection target is a license plate N, but of course, it is not limited to this. In this embodiment, the detection unit 34 uses YOLO (You Look Only Once) as the object detection process, but of course, it is not limited to this.

[0030] The determination unit 35 determines whether the coordinate position of the object to be detected includes the edge of the first divided image. In this embodiment, based on the detection result of the detection unit 34, the determination unit 35 determines whether there is an object to be detected (in this embodiment, a license plate N) in each of the first divided images B1 to B6, and if an object to be detected exists, it determines whether that object is located at the edge of the first divided image (the division position, the side facing the adjacent first divided image) (see Figures 3(B) and 4).

[0031] In other words, the determination unit 35 determines whether the object to be detected is divided by the first division unit 32. The object to be detected being at the edge of the first divided image means that a portion of the object is missing due to the division position. The edge of the first divided image refers to the portion located at the edge of the coordinate system of the first divided image.

[0032] Furthermore, if the determination unit 35 determines that the object to be detected is at the edge of the first divided image, it detects (determines) the first divided image that is located at a position adjacent to (facing) that edge (or one side including that edge). In other words, as shown in Figure 3(A), since the first divided image B2 is at the edge of the image where the object to be detected N is divided at position B21, the determination unit detects the first divided image B1 that is adjacent to that edge (divided position B21).

[0033] If the determination unit 35 determines that the detection target includes the edge of the first divided image, the extraction unit 36 ​​extracts an auxiliary image containing the detection target from the second divided image divided by the second divided image 33.

[0034] In this embodiment, when the determination unit 35 determines that the detection target includes the edge of the first segmented image, the extraction unit 36 ​​extracts an auxiliary image (for example, the second segmented image B7) in which the detection target N is captured from the second segmented image which has been previously segmented by the second segmented unit 33.

[0035] In other words, when the determination unit 35 determines that the target N to be detected includes the edge of the first segmented image B2, the extraction unit 36 ​​extracts a second segmented image B7 from within the second segmented image that spans the first segmented image B2 including the edge and the first segmented image B1 located adjacent to it.

[0036] The reading unit 37 reads information about the target N from the auxiliary image extracted by the extraction unit 36. In other words, the reading unit 37 performs character recognition processing (character recognition of license plate N) from the auxiliary image.

[0037] The memory unit 38 is composed of a storage device such as a semiconductor memory or a hard disk drive. The memory unit 38 stores programs and calculation parameters for executing various functions described later in the first division unit 32, the second division unit 33, the detection unit 34, the determination unit 35, the extraction unit 36, and the reading unit 37, as well as a learning model that is referenced during object detection processing in the second division unit 33 and the detection unit 34. The memory unit 38 is not limited to being built into the image processing device 30, but may be a separate storage device from the image processing device 30, or it may be a cloud server that can be connected via a network.

[0038] The communication unit 39 consists of a communication module that transmits and receives information (information related to license plate N) between the control center and other locations. The communication method is not particularly limited and may be wired or wireless.

[0039] The control center is a traffic control center that manages traffic signals belonging to a control area, but it is not limited to this; for example, it may be a control device that comprehensively controls multiple traffic signals within an intersection.

[0040] (Image processing method) Next, a typical operation of the image processing device 30 configured as described above will be explained. Figure 5 is a flowchart of the object detection process.

[0041] First, the acquisition unit 31 acquires the image (sensor image) captured by the camera 10 (sensor device) (step 101). The captured image (sensor image G) includes a vehicle C traveling on the road, as shown in Figure 2.

[0042] Next, the first division unit 32 divides the sensor image G into predetermined sizes (step 102). In this embodiment, the sensor image G is divided into 6 sections as shown in Figure 3(A).

[0043] Next, the second division section 33 divides the sensor image G so that it straddles the division position of the first divided image (step 103). In this embodiment, the sensor image G is divided into four parts by the second division section 33 (see Figures 3(B) and 4).

[0044] Next, the determination unit 35 determines whether there is a detection target in each of the first divided images (step 104). In this embodiment, the determination unit 35 determines whether the license plate N is in the sensor image G based on the detection result of the detection unit 34. If the license plate N is in the sensor image G (YES in step 104), the process proceeds to step 105.

[0045] Next, the determination unit 35 determines whether the object to be detected is at the edge of the first divided image (step 105). In this embodiment, as shown in Figure 3(A), for example, it is determined that the object to be detected (license plate N) in the first divided image B2 is at the edge (dividing position B21).

[0046] Next, if the determination unit 35 determines that the detection target is at the edge of the first segmented image B2 (YES in step 105), the extraction unit 36 ​​extracts the auxiliary image B7 (step 106). In this embodiment, the extraction unit 36 ​​extracts the detected second segmented image B7 based on the coordinate position of the edge of the first segmented image B2. In this embodiment, the first segmented image B2 was used as an example, but of course, it is not limited to this, and the first segmented images B1, B4, and B5 may also be used.

[0047] Next, the reading unit 37 reads information about the object to be detected from the auxiliary image B7 extracted by the extraction unit 36 ​​(step 107). In this embodiment, the reading unit 37 reads the character information of the license plate N from the auxiliary image.

[0048] Furthermore, if the determination unit 35 determines that the object to be detected is not at the edge of the first divided image (NO in step 105), the reading unit 37 reads information about the object to be detected from the first divided images B1 to B6 (step 108).

[0049] As described above, according to this embodiment, the image processing device 30 can efficiently detect the target object.

[0050] Traditionally, when using YOLO for object detection, the image was automatically scaled down to the YOLO input size (640x640) before detection. However, if the image could not be recognized after scaling down, it was split to match the YOLO input size.

[0051] In other words, conventionally, in order to perform recognition processing on an image with an input size (1920 x 1080) as shown in Figure 2 using YOLO, it was necessary to process all 10 images (to supplement the division positions) as shown in Figure 3(B).

[0052] However, according to this embodiment, first, the license plate N to be detected is detected from the first divided images B1 to B6 divided by the first division unit 32, and if the license plate N is at the edge of the first divided images B1 to B6, an auxiliary image is extracted from the second divided image based on the position of that edge. As a result, it is not necessary to perform recognition processing on all of the first and second divided images (up to 7 in this embodiment), so object detection can be performed efficiently (at high speed).

[0053] <Example 1> In the above description, the first division section 32 divided the image into predetermined sizes, but of course, this is not the only option. The size to be divided may be determined based on the position of the vehicle C in the sensor image G. Figure 6 is a block diagram showing an image processing system 100' according to a modified example, and Figure 7 is a diagram showing an example of division by the first division section 32 according to the modified example.

[0054] In this modified example, the image processing device 30' further includes a preprocessing unit 31'. The preprocessing unit 31' determines the location of the license plate N or the vehicle C (object) containing the license plate N, which are the object to be detected, within the sensor image G' acquired by the acquisition unit 31.

[0055] In other words, the pre-processing unit 31' determines the position of the vehicle C (or the object to be detected) within the sensor image G', and based on the determination result, the size to be divided by the first division unit 32 is determined.

[0056] For example, as shown in Figure 7, if the preprocessor 31' determines that vehicle C is traveling in the center of the sensor image G' in the left-right direction, the first division unit 32 divides the sensor image G' into four parts (first divided images B1' to B4').

[0057] In this case, the first division section 32 may be divided into four parts according to the input size of YOLO (i.e., the left and right edges of the sensor image G' are cut off), or the entire sensor image G' may be divided into four parts.

[0058] This allows for even faster object detection. In other words, if vehicle C is located at the center of sensor image G', the license plate N is likely to be located near the center of sensor image G'. Therefore, if an image of the central part of sensor image G' is available, the license plate N can be detected. Thus, it is sufficient to divide the sensor image G' so that the central part fits within the size. This reduces the number of divisions by the first division unit 32, thereby reducing the processing time required for object detection (up to 5 images in this embodiment). Also, in this modified example, the center includes the approximate center.

[0059] <Modification 2> In the above description, after dividing the image into predetermined sizes by the first division unit 32, a second division unit 33 generates a second divided image, and if the determination unit 35 determines that the object to be detected is at the edge of the first divided image, an auxiliary image is extracted from the previously generated second divided image. However, this is not the only option, and if the determination unit 35 determines that the object to be detected is at the edge of the first divided image, the second division unit 33 may generate an auxiliary image (second divided image) that includes the object to be detected.

[0060] In other words, the second division unit 33 generates a second division image that spans across the edge of the first division image detected by the determination unit 35 and the first division image located adjacent to it.

[0061] This eliminates the need to pre-generate multiple divided images using the second division unit 33, thereby shortening the processing time for object detection.

[0062] Furthermore, although the above embodiments have described the process of detecting objects (mainly vehicles) from captured images of roads as an example, the present invention is of course not limited to this, and can be applied to various monitoring systems in transportation facilities and commercial facilities. [Explanation of symbols]

[0063] 10...Sensor device 20…Machine learning device 30…Image processing device 31…Acquisition part 32...First division 33...Second division 34...Detection unit 35…Judgment section 36…Extraction part 37...Reading section G... Sensor image N... License plate

Claims

1. An acquisition unit that acquires sensor images generated by a sensor device, A first division unit divides the acquired sensor image into predetermined sizes, A second division section divides the sensor image so as to straddle the division position of the sensor image divided by the first division section, A detection unit for detecting a target within the first divided image divided by the first division unit, A determination unit that determines whether the coordinate position of the object to be detected includes the edge of the first divided image, If it is determined that the end portion is included, an extraction unit extracts an auxiliary image containing the detection target from the second divided image divided by the second division unit. An image processing apparatus comprising the following:

2. An image processing apparatus according to claim 1, The second division unit generates a second division image that spans the first division image before the determination unit determines that it includes the end portion. Image processing device.

3. An image processing apparatus according to claim 1, The second division portion is divided to include the detection target after it has been determined by the determination unit to include the end portion. Image processing device.

4. An image processing apparatus according to claim 1, If the determination unit determines that the end is included, it further determines the first divided image adjacent to the end, The extraction unit extracts a first segmented image including the end, and a second segmented image that spans the first segmented image adjacent to the end. Image processing device.

5. An image processing apparatus according to claim 1, The system further comprises a reading unit that reads information about the detection target from the target image extracted by the extraction unit. Image processing device.

6. A sensor device that generates a sensor image including the detected target, An image processing apparatus comprising: an acquisition unit for acquiring the sensor image generated by the sensor device; a first division unit for dividing the acquired sensor image into predetermined sizes; a second division unit for dividing the sensor image so as to straddle the division positions of the sensor image divided by the first division unit; a detection unit for detecting a target to be detected within the first divided image divided by the first division unit; a determination unit for determining whether the coordinate position of the target to be detected includes the edge of the first divided image; and, if it is determined that the edge is included, an extraction unit for extracting an auxiliary image including the target to be detected from the second divided image divided by the second division unit. An image processing system equipped with the following features.

Images