IMAGE PROCESSING SYSTEM, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM

Abstract

To provide an image processing system capable of precisely associating a moving body to be tracked among multiple images.SOLUTION: A disclosed image processing system 100 includes: an image acquisition unit that acquires multiple images including images of a moving body; an imaging direction information calculation unit that calculates imaging direction information representing the imaging direction from an imaging device to the moving body when capturing multiple images; a feature value calculation unit that calculates the feature value of the image of the moving body extracted from the multiple images; and an associating unit that associates the moving body among multiple images based on the imaging direction information and the feature value.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image processing system, an image processing method, and an image processing program. [Background technology]

[0002] There is known a technology for capturing images of moving objects such as vehicles and tracking the behavior of each moving object over a relatively wide area. As a related technology, Patent Document 1 discloses a technology for monitoring vehicles in a monitoring area using multiple behavior detection units. The behavior detection unit disclosed in Patent Document 1 stores images of multiple vehicles detected from captured images and ID numbers assigned to distinguish these vehicles. When a detected vehicle leaves the vehicle detection area, each behavior detection unit stores the information in a buffer. Other adjacent behavior detection units compare the captured image of each vehicle with the image of the vehicle in the buffer of the previous behavior detection unit, and assign an ID number to the vehicle based on the comparison result. By assigning the same ID number to the same vehicle, it becomes possible to track the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 10-105690 Summary of the Invention [Problem to be solved by the invention]

[0004] The behavior detection unit disclosed in Patent Document 1 compares multiple captured images and assigns an ID number to a vehicle based on the results. In such a case, if the posture of a moving object such as a vehicle changes or if there is a change in the lighting in the monitored area, there is a possibility that the same vehicle will be erroneously recognized as a different vehicle.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide an image processing system, an image processing method, and an image processing program capable of matching a moving object to be tracked between multiple images with high accuracy. [Means for solving the problem]

[0006] The image processing system according to the present disclosure comprises: an image acquisition unit that acquires a plurality of images including a moving object image; an imaging direction information calculation unit that calculates imaging direction information indicating an imaging direction from an imaging device to a moving object when the plurality of images are captured; a feature amount calculation unit that calculates a feature amount of the moving object image extracted from the plurality of images; The image capturing apparatus further includes a matching unit that matches the moving objects between the plurality of images based on the imaging direction information and the feature amount.

[0007] The image processing system according to the present disclosure comprises: an image acquisition unit that acquires a plurality of images including a moving object image; a feature amount calculation unit that calculates a feature amount of the moving object image extracted from the plurality of images and a feature amount of another moving object image extracted in the same image as the image including the moving object image; The image processing device further includes a matching unit that matches the moving object between the plurality of images based on a feature amount of the moving object image and a feature amount of the other moving object image.

[0008] The image processing method according to the present disclosure includes: An image acquisition step of acquiring a plurality of images including a moving object image; an imaging direction information calculation step of calculating imaging direction information indicating an imaging direction from an imaging device to a moving object when the plurality of images are captured; a feature amount calculation step of calculating a feature amount of the moving object image extracted from the plurality of images; and a matching step of matching the moving objects between the plurality of images based on the imaging direction information and the feature amount.

[0009] The image processing method according to the present disclosure includes: An image acquisition step of acquiring a plurality of images including a moving object image; a feature amount calculation step of calculating a feature amount of the moving object image extracted from the plurality of images and a feature amount of another moving object image extracted in the same image as the image including the moving object image; and a matching step of matching the moving objects between the plurality of images based on the feature amount of the moving object image and the feature amount of the other moving object image.

[0010] The image processing program according to the present disclosure comprises: An image acquisition step of acquiring a plurality of images including a moving object image; an imaging direction information calculation step of calculating imaging direction information indicating an imaging direction from an imaging device to a moving object when the plurality of images are captured; a feature amount calculation step of calculating a feature amount of the moving object image extracted from the plurality of images; a matching step of matching the moving objects between the plurality of images based on the imaging direction information and the feature amount; to be executed by the computer.

[0011] The image processing program according to the present disclosure comprises: An image acquisition step of acquiring a plurality of images including a moving object image; a feature amount calculation step of calculating a feature amount of the moving object image extracted from the plurality of images and a feature amount of another moving object image extracted in the same image as the image including the moving object image; a matching step of matching the moving object between the plurality of images based on a feature amount of the moving object image and a feature amount of the other moving object image; to be executed by the computer. Effect of the Invention

[0012] The image processing system, image processing method, and image processing program according to the present disclosure enable a moving object to be tracked to be associated with each other with high accuracy among a plurality of images. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a configuration of an image processing system according to a first embodiment. [Diagram 2] 4 is a flowchart showing a process of the image processing system according to the first embodiment. [Diagram 3] FIG. 1 is an explanatory diagram of an image processing system according to a first embodiment. [Figure 4] FIG. 1 is an explanatory diagram of an image processing system according to a first embodiment. [Diagram 5] FIG. 2 is an explanatory diagram of a case where imaging directions are similar in the image processing system according to the first embodiment. [Figure 6] FIG. 4 is an explanatory diagram for a case in which the imaging directions are not similar in the image processing system according to the first embodiment. [Figure 7] FIG. 11 is an explanatory diagram of an image processing system according to a second embodiment. [Figure 8] 10 is a flowchart showing a process of an image processing system according to a second embodiment. [Figure 9] 11 is a flowchart showing a process of identifying nearby vehicles in the image processing system according to the second embodiment. [Figure 10] FIG. 11 is an explanatory diagram of an image processing system according to a second embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] <Embodiment 1> An image processing system 100 according to the present embodiment will be described. The image processing system 100 is an information processing system that acquires a plurality of images including a moving object to be tracked, and associates the moving object between the plurality of images. The moving object may include, for example, a vehicle, a person, an animal, or a robot. The present disclosure may be applied to various moving objects that may be the target of tracking, without being limited to these.

[0015] In this embodiment, a case will be described in which the moving object to be tracked is a vehicle 200 traveling on a road. The image processing system 100 acquires a plurality of images captured by a camera installed on the road. The image processing system 100 extracts an image of the vehicle 200 from the acquired plurality of images, and associates the vehicle 200 among the plurality of images.

[0016] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. 1 is a block diagram showing the configuration of an image processing system 100 according to this embodiment. The image processing system 100 includes a camera CAM1, a camera CAM2, an image acquisition unit 101, a camera information acquisition unit 102, an object detection unit 103, an object feature extraction unit 104, a comparison unit 105, an ID output unit 106, a detection model 107, a feature extraction model 108, a comparison model 109, and a learning unit 110.

[0017] The cameras CAM1 and CAM2 are imaging devices that capture images of the vehicle 200 to be tracked. In the following description, the camera CAM1 or CAM2 may be simply referred to as a "camera". The camera is installed in an area where it is possible to capture an image of the vehicle 200. The camera may be, for example, an RGB camera, a monochrome camera, a grayscale camera, or an IR camera. Not limited to this, various cameras may be used. The number of cameras may be one or more. Here, the description will be given using two cameras, but three or more cameras may be used. In addition, these may be a combination of different types of cameras. For example, the camera CAM1 may be an RGB camera, and the camera CAM2 may be a monochrome camera.

[0018] The captured image obtained from the camera may be a still image or a moving image. The imaging direction of the camera may or may not be fixed. For example, the camera may be capable of imaging in a 360-degree direction and may capture images in different directions at predetermined time intervals.

[0019] The cameras are installed in an area where they can capture an image of a moving object to be tracked. In this embodiment, since the tracking object is a vehicle 200, the camera is assumed to be installed on a road on which the vehicle 200 travels. The image processing system 100 and the multiple cameras may be connected by wire or wirelessly. In FIG. 1, the cameras CAM1 and CAM2 are included in the image processing system 100, but they do not need to be mounted in the same device. For example, the cameras CAM1 and CAM2 may be installed on a road, and other functional units may be provided in a specified server.

[0020] The captured image may include something other than the moving object to be monitored. For example, a moving object other than the moving object to be monitored may be included in the captured image. In the following, a moving object other than the moving object to be monitored may be referred to as "another moving object". The other moving object may be, for example, a vehicle traveling in the same direction as the vehicle to be monitored. Without being limited thereto, the other moving object may be a person, an animal, a robot, or the like.

[0021] The image acquisition unit 101 acquires a plurality of captured images including an image of a moving object to be tracked from the camera. Here, the image acquisition unit 101 acquires captured images including a vehicle 220 from the cameras CAM1 and CAM2. The image acquisition unit 101 may acquire a captured image every time an image is captured by the camera, or may acquire captured images at a predetermined time interval. The image acquisition unit 101 acquires, together with the captured image, image capture date and time information indicating the date and time when the captured image was captured.

[0022] The camera information acquisition unit 102 acquires camera parameters, and outputs the acquired camera parameters to the object feature extraction unit 104 .

[0023] The object detection unit 103 uses the detection model 107 to detect the area of ​​an object included in the captured image acquired by the image acquisition unit 101 .

[0024] The object feature extraction unit 104 functions as an imaging direction information calculation unit and a feature amount calculation unit. The imaging direction information calculation unit is a functional unit that calculates imaging direction information indicating the imaging direction from the camera to the moving object during imaging. The feature amount calculation unit is a functional unit that calculates the feature amount of the moving object image extracted from multiple images. Each function will be described below. The detailed procedure of each process will be described later.

[0025] The object feature extraction unit 104 (imaging direction information calculation unit) calculates imaging direction information indicating the imaging direction from the camera to the moving object when capturing the multiple images captured by the image capture unit 101. The imaging direction information is, for example, the imaging direction from each of the cameras CAM1 and CAM2 to the vehicle 200, expressed as a unit direction vector n 1 and n 2 The object feature extraction unit 104 can be expressed as follows: 1 and n 2 In the following, the arrows on the letters indicating vectors may be omitted, and only the term vector may be used to indicate that the vector is a vector.

[0026] Here, the unit direction vector n 1 and n 2 Although it is explained that both are generated, they do not need to be generated simultaneously. 1 and n 2 may be generated at different times. For example, the generated unit direction vector n 1 or n 2 The image vector i 1 and i 2 , the integration vector g 1 and g 2 , feature vector f 1 and f 2The same applies to.

[0027] In addition, the object feature extraction unit 104 (feature amount calculation unit) calculates the feature amount of the moving object image detected by the object detection unit 103. Specifically, the object feature extraction unit 104 calculates the feature amount of the image of the vehicle 200 detected from the images captured by the cameras CAM1 and CAM2 in the unit direction vector n 1 and n 2 Each feature is calculated by taking into account the feature vector f 1 and f 2 The object feature extraction unit 104 performs the following process to obtain the feature vector f 1 and f 2 Generate.

[0028] First, the object feature extraction unit 104 extracts image vectors i 1 and i 2 Calculate each of the image vectors i 1 and i 2 may indicate the pixel value of each pixel in each image.

[0029] Next, the object feature extraction unit 104 extracts the unit direction vector n 1 and image vector i 1 and integrate the integration vector g 1 Similarly, the object feature extraction unit 104 generates a unit direction vector n 2 and image vector i 2 and integrate the integration vector g 2 Then, the object feature extraction unit 104 generates an integrated vector g 1 and g 2 From the feature vector f 1 and f 2 respectively.

[0030] In this manner, the object feature extraction unit 104 extracts a feature vector f 1and a feature vector f 2 and can be generated.

[0031] The object feature extraction unit 104 may calculate feature amounts of other moving object images extracted in the same image as the image including the vehicle 200. For example, when the vehicle 200 and a vehicle 210 different from the vehicle 200 are included in the same image, the object feature extraction unit 104 may calculate the feature amounts of the vehicle 200 and the vehicle 210 by the above-mentioned processing.

[0032] The comparison unit 105 and the ID output unit 106 function as a matching unit. The matching unit is a functional unit that matches moving objects between multiple images based on the image capture direction information and the feature amount of the captured image. Specifically, the comparison unit 105 matches the feature vector f generated by the object feature extraction unit 104 with the feature vector f generated by the object feature extraction unit 104. 1 and f 2 are compared using a comparison model 109. The ID output unit 106 assigns a moving object ID to the vehicle 200 included in the image based on the comparison result in the comparison unit 105. Here, the moving object ID is identification information for identifying the moving object included in the image. In this way, it is possible to associate the vehicles 200 between a plurality of images.

[0033] In the above comparison, the comparator 105 compares the unit direction vector n 1 and n 2 Different correspondences may be made depending on whether or not the unit direction vector n 1 and n 2 Whether or not they are similar may be determined by comparing them with a predetermined threshold value.

[0034] Moreover, the comparison unit 105 compares the unit direction vector n 1 and n 2 The comparison of the vehicles 200 may be performed using different criteria depending on whether the unit direction vectors n 1 and n 2are similar, a comparison is made using criteria including the shape of the vehicle 200, and the unit direction vector n 1 and n 2 If the vectors are not similar, the comparison may be performed using criteria that do not include the shape of the vehicle 200. 1 and n 2 If they are not similar, the comparison unit 105 may perform the comparison using a criterion that includes at least the color information of the moving object, for example.

[0035] The comparison unit 105 compares the unit direction vector n 1 and n 2 If they are similar, then the feature vector f 1 and f 2 If the images are not similar, the comparison unit 105 may not compare the feature vector f 1 and f 2 A comparison of the above may be performed.

[0036] The ID output unit 106 assigns a moving object ID to the extracted moving object image according to the result of the comparison in the comparison unit 105. When the comparison in the comparison unit 105 determines that the moving object is the same among the multiple images, the ID output unit 106 assigns the same moving object ID to the moving object image.

[0037] The learning unit 110 trains the detection model 107, the feature extraction model 108, and the comparison model 109.

[0038] Next, the process performed by the image processing system 100 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the process performed by the image processing system 100.

[0039] First, the image acquisition unit 101 acquires a captured image from a camera (S101). The captured image includes image capture date and time information indicating the date and time when the image was captured.

[0040] The object detection unit 103 performs extraction and identification processing of the input image using the detection model 107. The object detection unit 103 extracts moving object areas such as vehicles, bicycles, and pedestrians in the image. Here, the object detection unit 103 extracts the image area of ​​the vehicle 200 (S103). The object detection unit 103 may extract the moving object area using a general object recognition technology such as a deep learning network.

[0041] The object feature extraction unit 104 calculates a direction vector of the vehicle from the camera parameters acquired from the camera information acquisition unit 102 and the object area extracted in step S103 (S105).

[0042] The object feature extraction unit 104 generates an integrated vector by integrating the image features of the object extracted in step S103 and the directional vector calculated in step S105. The object feature extraction unit 104 receives the generated integrated vector as an input and generates a feature vector using a feature extractor such as a neural network (S107).

[0043] The comparison unit 105 compares the feature vector of the target object with the feature vector of the past frame (S109). The comparison unit 105 determines whether the feature vector of the target object is the same as the feature vector of the past frame (S111). If it is determined that the feature vectors are the same (YES in S111), the ID output unit 106 assigns the same ID (S113) and ends the process.

[0044] If it is determined in step S111 that the feature vectors are not identical (NO in S111), the process proceeds to step S117. The comparison unit 105 determines whether or not there is another feature vector (S117). If it is determined that there is another feature vector (YES in S117), the process returns to step S109. If it is determined that there is no other feature vector (NO in S117), the process proceeds to step S115. In this case, the ID output unit 106 assigns a new ID (S115) and ends the process.

[0045] The above-mentioned processes in steps S105, S107, and S109 will be described in detail with reference to Fig. 3. Fig. 3 shows the unit direction vector n 1 , image vector i 1 , the integration vector g 1 , and feature vector f 1 1 is a diagram showing an overview of the process for generating a unit direction vector n 2 , image vector i 2 , the integration vector g 2、 and feature vector f 2 can be generated in a similar manner.

[0046] In the following explanation, the image vector i 1 or i 2 In some cases, we will refer to it simply as the "image vector i". Similarly, the unit direction vector n 1 or n 2 Let n be the unit direction vector and g be the integrated vector. 1 or g 2 Let g be the "integrated vector" and f be the feature vector. 1 or f 2 is sometimes referred to as the "feature vector f".

[0047] In step S105 described above, the unit direction vector n of the vehicle 200 is calculated from the camera parameters and the object region extracted in step S103 in the following procedure. Here, the camera parameters are expressed as a 3-row, 4-column projection matrix A. As shown in Fig. 3, the position of the center of gravity of the vehicle in the three-dimensional world coordinates is P(X,Y,Z), and the position of the center of gravity in the image coordinates is p(u,v).

[0048] The relationship between P(X,Y,Z) and p(u,v) is expressed by the following equation (1).

[0049]

number

[0050] In formula (1), S is a scale parameter. Furthermore, the projection matrix A is a known matrix obtained by calibrating the camera CAM1. Furthermore, the projection matrix A can be expressed by the following formulas (2) and (3).

[0051]

number

[0052]

number

[0053] From equations (1) to (3), the following equation (4) is obtained.

[0054]

number

[0055] In step S105, the unit direction vector n of the vehicle 200 is calculated as shown in the following equation (5).

[0056]

number

[0057] In equation (5), the direction vector N is expressed by the following equation (6).

[0058]

number

[0059] In step S107, the feature vector f is calculated in the following manner. First, detection frame D of vehicle 200 shown in FIG. 3 is normalized to a rectangular area of ​​K*K. Detection frame D may be an image area detected by object detection unit 103. Next, raster scanning is performed on the normalized rectangular area. Then, pixel values ​​of the rectangular area arranged in raster scan order are defined as image vector i as shown in the following equation (7). Here, the number of dimensions is equal to the number of pixels in the normalized rectangular area.

[0060]

number

[0061] Next, the unit direction vector n calculated in (5) of step S105 and the image vector i defined in equation (7) are integrated to generate an integrated vector g expressed by the following equation (8).

[0062]

number

[0063] The integrated vector g generated by equation (8) is input to the learning device of the feature extraction model 108, and a feature vector f is obtained as an output. An example of the structure of the feature extraction model 108 and the comparison model 109 is shown in FIG. 4. In FIG. 4, the integrated vector g 2 and the integrated vector g of the past frame 1 is shown.

[0064] As the learner, various machine learning models can be used, for example, a deep learning convolutional neural network (CNN).

[0065] In step S109, the feature vector f of the target object obtained in step S107 is 2 and the feature vector f 1 The comparison unit 105 compares the feature vector f 1 and f 2If the output is 1, it is determined that the two are identical, and if it is 0, they are not identical. As the learning device, various machine learning models can be used. For example, a hierarchical neural network can be used.

[0066] Various general-purpose CNNs can be used as the feature extraction model 108. For example, VGG, EfficientNet, ResNet, etc. can be used. Also, a general-purpose fully-connected multi-layer perceptron can be used as the comparison model 109. The feature extraction model 108 and the comparison model 109 are trained simultaneously.

[0067] The training data is a set of data expressed by the following formula (9) and a label expressed by the following formula (10).

[0068]

number

[0069]

number

[0070] The integration vector g has already been defined in equation (8). In equation (10), id t is the mobile ID of the mobile at time t. 1 and g 2 For input, it learns to input 1 if the mobile ID is equal and 0 if they are different.

[0071] In this manner, in this embodiment, the unit direction vector n from the camera is added to the image vector i of the moving object image to generate a feature vector f to be used for image comparison. Therefore, moving object images can be matched more accurately than when only the image vector i is used for comparison.

[0072] 5 and 6 are diagrams showing a schematic diagram of the processing of the image processing system 100 according to the present embodiment. As shown by the outline arrow, the vehicle 200 is traveling from point A1 on the right side of the paper to point A2 on the left side of the paper. The vehicle 200 is imaged by the cameras CAM1 and CAM2 at points A1 and A2, respectively.

[0073] In Fig. 5, cameras CAM1 and CAM2 are arranged to capture images of vehicle 200 from the same direction. In the figure, cameras CAM1 and CAM2 capture images of vehicle 200 from the front. In contrast, in Fig. 6, cameras CAM1 and CAM2 are arranged back-to-back. Camera CAM1 captures images of vehicle 200 from the front, and camera CAM2 captures images of vehicle 200 from the rear. Therefore, the shape, etc. of vehicle 200 differs between the images captured at points A1 and A2.

[0074] In the image processing system 100 according to this embodiment, images are compared using different criteria between the case shown in Fig. 5 and the case shown in Fig. 6, and the vehicles 200 can be matched. For example, in the case shown in Fig. 5, the matching is performed using a criterion including the shape of the vehicle 200. The criterion including the shape of the vehicle 200 may be a criterion using features such as the shape of the vehicle 200 or texture (pixel value of each pixel).

[0075] 6, images can be compared and matched using a criterion that does not include the shape of the vehicle 200. The criterion that does not include the shape of the vehicle 200 may be a criterion that uses color features of the vehicle 200. For example, the criterion may be a criterion that uses features such as the overall color of the vehicle 200 (e.g., white) or the pattern of the vehicle 200. By performing comparison using such features that do not depend on the camera orientation, the vehicles 200 can be matched with high accuracy.

[0076] 6, when the imaging directions of the cameras CAM1 and CAM2 are significantly different, the images captured by the camera CAM2 may not be used for matching the images. In this way, it is possible to compare only images captured in similar directions, and therefore to perform matching that weights the characteristics related to the shape of the vehicle 200.

[0077] 5 and 6 show an example in which the two cameras capture images of the vehicle 200 from the same direction (front) and an example in which the cameras capture images from opposite directions (front or rear), but this is not limited to this. The image capturing direction may include any direction. For example, it may include an image capturing direction from the side or top of the vehicle 200. Therefore, it may include cases in which the image capturing directions differ by 90 degrees. Different correspondences may be made depending on whether the image capturing directions are similar. Whether the image capturing directions are similar is determined by calculating a unit direction vector n 1 and n 2 This may be done by comparing it with a predetermined threshold value that is set in advance.

[0078] Furthermore, the image processing system 100 is not limited to this, and may perform any weighting to associate the vehicle 200.

[0079] As described above, in the image processing system 100 according to this embodiment, the image acquisition unit 101 acquires a plurality of images including the vehicle 200 from the camera. In addition, the object feature extraction unit 104 calculates imaging direction information (unit direction vector n) indicating the imaging direction from the camera to the moving object, and feature amounts (image vector i) of each moving object image extracted from the plurality of images. The object feature extraction unit 104 calculates an integrated vector g from the unit direction vector n and the image vector i, and calculates a feature vector f from the integrated vector g. The comparison unit 105 compares the feature vectors f between images captured at different times. The ID output unit 106 associates the images by assigning the same moving object ID to the same vehicle 200 based on the comparison result in the comparison unit 105.

[0080] With this configuration, the image processing system 100 can perform correspondence based on color information that does not depend on the camera's attitude when the unit direction vector n is different, that is, when the shape features of the vehicle 200 in the captured image are considered to be different. Therefore, the image processing system 100 according to this embodiment can learn an optimal ID inference engine according to the arrangement of the cameras. This makes it possible to prevent erroneous recognition of the captured moving object and to track the moving object with high accuracy.

[0081] <Embodiment 2> Next, a second embodiment of the present disclosure will be described. In the first embodiment, the moving objects are associated among the multiple images based on the imaging direction information (direction vector) calculated by the object feature extraction unit 104 (imaging direction information calculation unit) and the feature amount (image vector) of each captured image. In the present embodiment, instead of the above-mentioned direction vector, the association is performed by taking into account information of a moving object other than the vehicle 200 extracted in the captured image.

[0082] 7 is a diagram showing a schematic diagram of the processing of the image processing system 100 according to the present embodiment. As shown in the figure, at a point A1, vehicles 210 and 220 are present near a vehicle 200 and are traveling in the same direction. Here, it is vehicle 200 that is the subject of matching between a plurality of images, and vehicles 210 and 220 are not the subject of matching.

[0083] The configuration of the image processing system 100 according to this embodiment is similar to the block diagram shown in Fig. 1. The configuration of the image processing system 100 according to this embodiment will be described with reference to Fig. 1. Description of the same configuration as in the first embodiment will be omitted.

[0084] The image acquisition unit 101 acquires a plurality of captured images including moving object images from the cameras. Here, the captured images include moving objects other than the vehicle 200. Here, as shown in Fig. 7, it is assumed that the image captured by the camera CAM1 includes the vehicles 200, 210, and 220 in the same image. It is also assumed that the image captured by the camera CAM2 includes the vehicles 200 and 210 in the same image, but does not include the vehicle 220 in the same image.

[0085] The object detection unit 103 detects the object regions of the vehicles 200 to 220 from the image acquired by the image acquisition unit 101 using the detection model 107 .

[0086] The object feature extraction unit 104 functions as a feature amount calculation unit. The object feature extraction unit 104 calculates the feature amount of each image of the vehicles 200 to 220 extracted by the object detection unit 103. As in the first embodiment, the object feature extraction unit 104 generates an image vector i indicating the feature amount of each image.

[0087] Here, the object feature extraction unit 104 may specify a vehicle having a moving speed close to that of the vehicle 200, which is the target of the association, among the vehicles 210 and 220 that are not the target of the association, and calculate the image vector i. The moving speed of the vehicles 200 to 220 may be calculated based on a plurality of images captured by the camera CAM1. For example, the moving speed may be calculated based on the image capture interval and the moving distance of each of the vehicles 200 to 220 by comparing a plurality of images captured by the camera CAM1 at a predetermined time interval. The speed of each vehicle may be calculated using other calculation methods without being limited to this.

[0088] In FIG. 7, the traveling speeds of the vehicles 200, 210, and 220 at the point A1 are represented by the velocity vectors V i , the velocity vector V j , and the velocity vector V KIt is assumed that the vehicle 200 and the vehicle 210 are traveling at a relatively similar speed, and the vehicle 220 is traveling at a relatively faster speed than the vehicle 200. The object feature extraction unit 104 generates an image vector i of an image corresponding to the vehicle 210, which has a similar traveling speed to the vehicle 200.

[0089] The object feature extraction unit 104 may determine whether the moving speeds are close to each other by setting a predetermined threshold value in advance. The object feature extraction unit 104 may determine that the moving speeds are close to each other when the difference between the moving speed of the vehicle 200 and the moving speed of another vehicle is equal to or less than a predetermined value. In the following description, a vehicle having a moving speed close to that of the vehicle 200 may be referred to as a "nearby vehicle."

[0090] There may be a plurality of nearby vehicles. Priorities may be set for the plurality of nearby vehicles and used for matching with the vehicle 200. For example, the priority may be increased in the order of the moving speed of the nearby vehicles being closer to the vehicle 200, and the matching may be weighted according to the priority.

[0091] 1, the description will be continued. The comparison unit 105 and the ID output unit 106 have a function of a matching unit. The matching unit matches the vehicle 200 between a plurality of images based on the feature amount of the image of the vehicle 200 and the feature amount of the image of the vehicle 200.

[0092] Specifically, the comparison unit 105 compares the multiple images by comparing the feature vectors of the vehicle 200 generated by the object feature extraction unit 104. The comparison unit 105 may obtain the comparison result by using a comparison model 109. The ID output unit 106 assigns a moving object ID according to the comparison result in the comparison unit 105.

[0093] Next, the process performed by the image processing system 100 according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the process performed by the image processing system 100 according to this embodiment.

[0094] First, the image acquisition unit 101 acquires a captured image from a camera (S201). The captured image includes image capture date and time information indicating the date and time of capture. The object detection unit 103 performs extraction and identification processing of the input image using the detection model 107, and extracts moving object areas such as vehicles, bicycles, and pedestrians in the image (S203). The object detection unit 103 may extract the moving object areas using a general object recognition technology such as a deep learning network.

[0095] The object feature extraction unit 104 calculates an image vector of a nearby vehicle included in the captured image (S205). Here, the process of identifying a nearby vehicle will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the process of identifying a nearby vehicle.

[0096] The object feature extraction unit 104 adds vehicles (hereinafter referred to as "nodes") whose moving speeds are similar to those of the vehicle 200 to the same grouping graph 300, and deletes nodes whose moving speeds are not similar to those of the vehicle 200 from the grouping graph 300. The object feature extraction unit 104 adds or deletes nodes by the following process.

[0097] First, the object feature extraction unit 104 determines the vehicle 200, which is the target of the association, as a root node (S301). The object feature extraction unit 104 determines candidate nodes to be grouped with the vehicle 200 (S303). For example, the object feature extraction unit 104 determines vehicles that are within a predetermined distance from the center of gravity of the orthoimage of the target vehicle as candidate nodes. In the example shown in FIG. 7, vehicles 210 and 220 are candidate nodes. Without being limited to this, the candidate nodes may be determined using any condition. For example, all moving objects included in the captured image may be determined as candidate nodes.

[0098] The object feature extraction unit 104 determines, from among the candidate nodes, a vehicle whose absolute value of the difference between the speed vector and the vehicle 200 is equal to or less than a predetermined value, as a node in the same grouping graph 300 as the vehicle 200 (S305). The edge strength is the reciprocal of the absolute value of the difference. Here, the object feature extraction unit 104 determines the vehicle 210 as a node in the same grouping graph 300.

[0099] The object feature extraction unit 104 deletes any of the candidate nodes that do not satisfy the condition in step S305 (S307). In this example, the vehicle 220 is deleted.

[0100] In this manner, the object feature extraction unit 104 can identify nearby vehicles that belong to the same group as the vehicle 200. Here, the object feature extraction unit 104 identifies the vehicle 210 as a nearby vehicle of the vehicle 200.

[0101] Returning to Fig. 8, the description will continue. The object feature extraction unit 104 generates feature vectors using the image vectors of the vehicles 200 and 210 (S207). The method of generating each feature vector is the same as in the first embodiment, so a detailed description will be omitted.

[0102] Fig. 10 corresponds to Fig. 4 in the first embodiment, and shows an example of the structure of the feature extraction model 108 and the comparison model 109. The object feature extraction unit 104 generates an image vector i of the vehicle 200 and an image vector j of the vehicle 210 in the same manner as in equation (7) in the first embodiment.

[0103] Next, the object feature extraction unit 104 integrates the image vectors i and j to generate an integrated vector g in the same manner as in equation (8) in the first embodiment. The integrated vector g is expressed by the following equation (11).

[0104]

number

[0105] The object feature extraction unit 104 generates a feature vector f from the integrated vector g in the same manner as in the first embodiment. 2 and the feature vector f 1 respectively.

[0106] The comparison unit 105 compares the feature vector f obtained in step S207 with 1 and f2 The feature vector f 1 and f 2 For input, it learns to input 1 if the mobile ID is equal and 0 if they are different.

[0107] The subsequent processing is the same as in the first embodiment, and therefore the description thereof will be omitted. The processing in steps S211 to S217 corresponds to steps S111 to S117 described with reference to FIG.

[0108] In addition, when there are multiple nearby vehicles, feature vectors may be generated for all the nearby vehicles, or the nearby vehicles used for matching may be limited and feature vectors for the limited nearby vehicles may be generated. For example, a priority order for matching may be set according to the moving speed of the vehicle 200, the positional relationship with the vehicle 200, etc.

[0109] As described above, according to the image processing system 100 of this embodiment, a feature vector is generated by adding image information of the vehicle 210 existing near the vehicle 200 to be associated. Also, the distance from the vehicle 200 and the difference in the moving speed between the vehicle 200 and the vehicle 210 are used to identify nearby vehicles.

[0110] In this way, by generating features by adding image information of nearby moving objects that can be regarded as being in the same group, it is possible to match images more accurately than when using image information of the vehicle 200 alone. Even if there are multiple vehicles that have shape features similar to the vehicle 200, it is possible to appropriately match the vehicle 200 between multiple images.

[0111] This embodiment may be realized by combining the configurations described in embodiment 1. The vehicle 200 may be associated using both the direction vector described in embodiment 1 and the image vector of the nearby vehicle described in this embodiment.

[0112] <Hardware configuration example> Each functional component of the image processing system 100 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of the image processing system 100 is realized by a combination of hardware and software.

[0113] 11 is a block diagram illustrating an example of a hardware configuration of a computer 900 that realizes the image processing system 100. The computer 900 may be a dedicated computer designed to realize the image processing system 100, or may be a general-purpose computer. The computer 900 may be a portable computer such as a smartphone or a tablet terminal.

[0114] For example, by installing a predetermined application on the computer 900, each function of the image processing system 100 is realized by the computer 900. The application is constituted by a program for realizing the functional components of the image processing system 100.

[0115] The computer 900 includes a bus 902, a processor 904, a memory 906, a storage device 908, an input / output interface 910, and a network interface 912. The bus 902 is a data transmission path for the processor 904, the memory 906, the storage device 908, the input / output interface 910, and the network interface 912 to transmit and receive data to and from each other. However, the method of connecting the processor 904 and other components to each other is not limited to a bus connection.

[0116] The processor 904 is one of various processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 906 is a main storage device realized using a random access memory (RAM) or the like. The storage device 908 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0117] The input / output interface 910 is an interface for connecting the computer 900 to an input / output device. For example, an input device such as a keyboard and an output device such as a display device are connected to the input / output interface 910. For example, the camera described in the embodiment may be connected to the input / output interface 910.

[0118] The network interface 912 is an interface for connecting the computer 900 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0119] The storage device 908 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the image processing system 100. The processor 904 reads this program into the memory 906 and executes it to realize each functional component of the image processing system 100.

[0120] Each of the processors executes one or more programs including instructions for causing a computer to perform an algorithm. The programs include instructions (or software code) for causing a computer to perform one or more functions described in the embodiments when loaded into the computer. The programs may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, a transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0121] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0122] 100 Image Processing System 101 Image acquisition unit 102 Camera information acquisition unit 103 Object detection unit 104 Object feature extraction unit 105 Comparison section 106 ID output section 107 Detection Model 108 Feature Extraction Model 109 Comparison Models 110 Learning Department 200, 210, 220 vehicles 300 Grouped Graphs CAM1, CAM2 cameras D Detection Frame N-directional vector n,n 1 , n 2 Unit Direction Vector i,i 1 , i 2 , j image vector g, g 1 , g 2 Integration Vector f, f 1 , f 2 Feature Vector V i , V j , V K Velocity Vector

Claims

1. an image acquisition unit that acquires a plurality of images including a moving object image; an imaging direction information calculation unit that calculates imaging direction information indicating an imaging direction from an imaging device to a moving object when the plurality of images are captured, the imaging direction information including a direction of the imaging device; a feature amount calculation unit that calculates a feature amount of the moving object image extracted from the plurality of images; a matching unit that matches the moving object between the plurality of images based on the imaging direction information and the feature amount; Equipped with the imaging direction information is a direction vector from the imaging device to the moving object, the direction vector including a direction of the imaging device; The association unit determines whether the direction vectors are similar between the plurality of images by comparing with a predetermined threshold, and performs different associations when the direction vectors are similar between the plurality of images and when the direction vectors are not similar between the plurality of images. Image processing system.

2. The matching unit matches the moving objects using different criteria when the direction vectors are similar between the plurality of images and when the direction vectors are not similar between the plurality of images. The image processing system according to claim 1 .

3. The association unit is If the direction vectors are similar between the plurality of images, the moving objects are associated with each other using a criterion including a shape of the moving object; If the direction vectors are not similar between the plurality of images, the moving objects are matched using a criterion that does not include the shape of the moving object.

3. The image processing system according to claim 1.

4. When the direction vectors are not similar between the plurality of images, the association unit associates the moving objects using a criterion including at least color information of the moving objects. The image processing system according to any one of claims 1 to 3.

5. an image acquisition unit that acquires a plurality of images including a moving object image; a feature amount calculation unit that calculates a feature amount of the moving object image extracted from the plurality of images and a feature amount of another moving object image extracted in the same image as the image including the moving object image; a matching unit that matches the moving object between the plurality of images based on a feature amount of the moving object image and a feature amount of the other moving object image; Equipped with the feature amount calculation unit calculates a feature amount of the image of the other moving object when a difference between a moving speed of the moving object and a moving speed of the other moving object is equal to or smaller than a predetermined value; The association unit determines whether or not the feature amounts of the other moving object images are similar among the multiple images by comparing with a predetermined threshold value, and performs different associations when the feature amounts of the other moving object images are similar among the multiple images and when the feature amounts of the other moving object images are not similar among the multiple images. Image processing system.

6. An image acquisition step of acquiring a plurality of images including a moving object image; an imaging direction information calculation step of calculating imaging direction information indicating an imaging direction from an imaging device to a moving object when the plurality of images are captured, the imaging direction information including a direction of the imaging device; a feature amount calculation step of calculating a feature amount of the moving object image extracted from the plurality of images; a matching step of matching the moving objects between the plurality of images based on the imaging direction information and the feature amount; Equipped with the imaging direction information is a direction vector from the imaging device to the moving object, the direction vector including a direction of the imaging device; In the association step, whether or not the direction vectors are similar between the plurality of images is determined by comparing with a predetermined threshold value, and different associations are performed when the direction vectors are similar between the plurality of images and when the direction vectors are not similar between the plurality of images. Image processing methods.

7. An image acquisition step of acquiring a plurality of images including a moving object image; a feature amount calculation step of calculating a feature amount of the moving object image extracted from the plurality of images and a feature amount of another moving object image extracted in the same image as the image including the moving object image; a matching step of matching the moving object between the plurality of images based on a feature amount of the moving object image and a feature amount of the other moving object image; Equipped with In the feature amount calculation step, when a difference between a moving speed of the moving object and a moving speed of the other moving object is equal to or smaller than a predetermined value, a feature amount of the image of the other moving object is calculated; In the matching step, whether or not the feature amounts of the other moving object images are similar among the multiple images is determined by comparing with a predetermined threshold value, and different matching is performed when the feature amounts of the other moving object images are similar among the multiple images and when the feature amounts of the other moving object images are not similar among the multiple images. Image processing methods.

8. An image acquisition step of acquiring a plurality of images including a moving object image; an imaging direction information calculation step of calculating imaging direction information indicating an imaging direction from an imaging device to a moving object when the plurality of images are captured, the imaging direction information including a direction of the imaging device; a feature amount calculation step of calculating a feature amount of the moving object image extracted from the plurality of images; a matching step of matching the moving objects between the plurality of images based on the imaging direction information and the feature amount; on the computer, the imaging direction information is a direction vector from the imaging device to the moving object, the direction vector including a direction of the imaging device; In the association step, whether or not the direction vectors are similar between the plurality of images is determined by comparing with a predetermined threshold value, and different associations are performed when the direction vectors are similar between the plurality of images and when the direction vectors are not similar between the plurality of images. Image processing program.

9. An image acquisition step of acquiring a plurality of images including a moving object image; a feature amount calculation step of calculating a feature amount of the moving object image extracted from the plurality of images and a feature amount of another moving object image extracted in the same image as the image including the moving object image; a matching step of matching the moving object between the plurality of images based on a feature amount of the moving object image and a feature amount of the other moving object image; on the computer, In the feature amount calculation step, when a difference between a moving speed of the moving object and a moving speed of the other moving object is equal to or smaller than a predetermined value, a feature amount of the image of the other moving object is calculated; In the matching step, whether or not the feature amounts of the other moving object images are similar among the multiple images is determined by comparing with a predetermined threshold value, and different matching is performed when the feature amounts of the other moving object images are similar among the multiple images and when the feature amounts of the other moving object images are not similar among the multiple images. Image processing program.

Images