Vehicle identification code determination device and method

Abstract

To improve the accuracy of estimating vehicle number plates. [Solution] The tracking means 75 outputs the tracking results of the vehicle body frame and license plate frame between the front and rear frames based on the vehicle body frame estimated by the first AI estimation means 71. The area linking means 76 links the vehicle body frame and the license plate frame in each frame based on rules and outputs the linking result data. The temporary identifier assignment means 77 assigns a temporary identifier to the vehicle body frame or license plate frame of the previous frame when tracking is possible based on the tracking results in other frames. The determination means 78 adjusts the numbers assigned the same temporary identifier in multiple frames using preset rules when numbers do not match.

Description

[Technical Field]

[0001] The present invention relates to a vehicle identification code determination device. [Background technology]

[0002] Patent Document 1 discloses a motorcycle that captures images of vehicles with a mounted camera and recognizes the license plate of a stolen or criminally committed vehicle from the images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-051561 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the inventor's experiments, there are various problems when reading a vehicle's license plate and identifying the number using artificial intelligence (AI), as in Patent Document 1. For example, 1) the image captured by the camera is unclear, making it impossible to identify; 2) the image is partially cut off due to a difference in the timing of the image capture (especially the license plate), making it impossible to identify; 3) when extracting the license plate area from the captured image, a different area may be mistakenly extracted, making it impossible to identify; and 4) the extracted license plate is inaccurately identified.

[0005] In particular, the above problem has a much greater impact when a moving vehicle is imaged using a camera mounted on it, compared to when the vehicle is imaged from a fixed position on the road.

[0006] SUMMARY OF THE INVENTION In order to alleviate the above-mentioned problems, an object of the present invention is to provide a vehicle identification code determination device that can detect a vehicle identification code with higher accuracy using video capture data.

[0007] Another object of the present invention is to provide a vehicle identification code determination device that can extract a vehicle outer shape area and a vehicle identification code area. [Means for solving the problem]

[0008] (1) A vehicle identification code determination device according to the present invention is a vehicle identification code determination device that, when given frame image data in which still images including a vehicle's outer shape region and / or a vehicle identification code region are arranged in a time series, estimates a character string in a vehicle identification code region of a certain vehicle from the frame image data, and includes: B) a first AI estimation means that, when given the frame image data as input data, estimates the vehicle outer shape region and vehicle identification code region in each frame image data and outputs them as vehicle outer shape region identification data and vehicle identification code region identification data, respectively; C) a second AI estimation means that extracts an image of the vehicle identification code region from the vehicle identification code region identification data estimated by the first AI estimation means and image data including the vehicle outer shape, and estimates the vehicle identification code; D) a tracking means that outputs the following vehicle outer shape region tracking result data and vehicle identification code region tracking result data based on the vehicle outer shape region estimated by the first AI estimation means and the image data of the vehicle outer shape region; d1) a vehicle identification code determination device that estimates a character string in a vehicle outer shape region of a certain vehicle from the frame image data in which still images including the vehicle's outer shape region are arranged in a time series; both outer peripheral area tracking result data; d2) vehicle identification code area tracking result data generated by object tracking of the image in the vehicle identification code area, which performs object tracking processing of the image in the vehicle identification code area; E) area linking means for linking the vehicle identification code area estimated by the first AI estimation means and the vehicle identification code area in each frame based on a predetermined decision rule, and outputting the linking result data as intra-frame linking result data; F) means for assigning a common vehicle temporary identifier to the vehicle identification code area and the vehicle identification code area when the vehicle identification code area is linked to the vehicle identification code area in the intra-frame linking result data of a specific frame of the time-series still image data, wherein when tracking of the vehicle outer peripheral area or the vehicle identification code area of ​​the specific frame is possible in at least one of the vehicle outer peripheral area tracking result data or the vehicle identification code area tracking result data in a frame other than the specific frame, temporary identifier assigning means for assigning the vehicle temporary identifier assigned to the vehicle outer peripheral area or the vehicle identification code area of ​​the specific frame as the assigned vehicle temporary identifier;G) A means for determining a vehicle identification code by associating the vehicle identification code estimated in each frame by the second AI estimation means with respect to the vehicle identification code area to which the vehicle temporary identifier is assigned, and includes a determination means for adjusting the vehicle identification code using a preset adjustment rule when the vehicle identification codes assigned to the same vehicle temporary identifier in multiple frames do not match.

[0009] In this way, the vehicle exterior area and the vehicle identification code area are linked for each frame, and the temporary vehicle identifier is assigned using the tracking results of the vehicle exterior area and the vehicle identification code area between frames. Furthermore, if tracking is possible using either the vehicle exterior area or the vehicle identification code area associated with the temporary vehicle identifier, the same temporary vehicle identifier is assigned. This allows vehicles to be linked between multiple frames. Furthermore, if the read vehicle identification code areas do not match, the vehicle identification code area can be determined by adjusting them. (2) A vehicle identification code determination device according to the present invention is a vehicle identification code determination device that, when given frame image data in which still images including a vehicle's outer shape region and / or vehicle identification code region are arranged in chronological order, estimates a character string in a vehicle identification code region of a certain vehicle from the frame image data, and includes: B) a first AI estimation means that, when given the frame image data as input data, estimates the vehicle outer shape region in each frame image data and the vehicle identification code region in that vehicle outer shape region, and outputs them as vehicle outer shape region identification data and vehicle identification code region identification data, respectively; C) a second AI estimation means that extracts an image of the vehicle identification code region from the vehicle identification code region identification data estimated by the first AI estimation means and image data including the vehicle outer shape, and estimates the vehicle identification code; and D) a tracking means for outputting the following vehicle exterior area tracking result data and vehicle identification code area tracking result data based on the vehicle exterior area estimated by the first AI estimation means and its image data; d1) vehicle exterior area tracking result data generated by performing object tracking processing on images in the vehicle exterior area between previous and next frames of the time-series still image data; d2) vehicle identification code area tracking result data generated by object tracking on images in the vehicle identification code area by performing object tracking processing on images in the vehicle identification code area; E) means for assigning a vehicle temporary identifier common to the vehicle identification code area and the vehicle identification code area in a specific frame of the time-series still image data when the vehicle identification code area exists within the vehicle identification code area, in a frame other than the specific frame,F) a temporary identifier assigning means for assigning the vehicle temporary identifier assigned to the vehicle external area or the vehicle specific code area of ​​the specified frame as the vehicle temporary identifier to be assigned when the vehicle external area or the vehicle specific code area of ​​the specified frame has been tracked in at least one of the vehicle external area tracking result data or the vehicle specific code area tracking result data; F) a means for determining a vehicle specific code by associating the vehicle specific code estimated in each frame by the second AI estimation means with the vehicle specific code area to which the vehicle temporary identifier has been assigned, and when the vehicle specific codes assigned with the same vehicle temporary identifier in multiple frames do not match, the means includes a determination means for adjusting using a preset adjustment rule.

[0010] In this way, the vehicle exterior area and the vehicle identification code area are linked for each frame, and the temporary vehicle identifier is assigned using the tracking results of the vehicle exterior area and the vehicle identification code area between frames. Furthermore, if tracking is possible using either the vehicle exterior area or the vehicle identification code area associated with the temporary vehicle identifier, the same temporary vehicle identifier is assigned. This allows vehicles to be linked between multiple frames. Furthermore, if the read vehicle identification code areas do not match, the vehicle identification code area can be determined by adjusting them.

[0011] (3) In the vehicle identification code determination device according to the present invention, for the vehicle identification code in the time-series still image data estimated by the second estimation means, it is determined whether the moving direction of the vehicle is an approaching direction or a non-approaching direction based on the positional relationship of each vehicle identification code area between the front and rear frames determined by the tracking means or the positional relationship of each vehicle identification code area, and if the moving direction of the vehicle is an approaching direction, the vehicle identification code area determined in the rear frame is adopted, and if the moving direction of the vehicle is a non-approaching direction, the vehicle identification code area determined in the front frame is adopted. Thus, the vehicle identification code area to be adopted can be determined based on the moving direction of the vehicle.

[0012] (4) When the character string represented by the vehicle identification code estimated by the second AI estimation means is unclear in one of the front and rear frames, the vehicle identification code determination device according to the present invention identifies the character string in the other frame different from the one frame as the vehicle identification code. In this way, by assigning a code indicating the unclearness, the accuracy rate can be improved.

[0013] (5) The vehicle identification code determination device according to the present invention determines that the vehicle identification code is composed of character groups "area name," "classification number," "hiragana," and "series designation number," and that each of the character groups is represented by the vehicle identification code estimated by the second AI estimation means. Therefore, a character string can be determined for each of the character groups.

[0014] (6) The vehicle identification code determination device according to the present invention determines the character string in one or the other frame as the character string that constitutes the vehicle identification code for each character in the character group, thereby determining the character string for each character in the character group.

[0015] (7) A vehicle identification code determination device according to the present invention is a vehicle identification code determination device that, when given frame image data in which still images including a vehicle's outer shape area and / or vehicle identification code area are arranged in chronological order, estimates a character string in the vehicle identification code area of ​​a certain vehicle from the frame image data, and includes: B) a first AI estimation means that, when given the frame image data as input data, estimates the vehicle outer shape area and vehicle identification code area in each frame image data and outputs them as vehicle outer shape area identification data and vehicle identification code area identification data, respectively; and C) a second AI estimation means that extracts an image of the vehicle identification code area from the vehicle identification code area identification data estimated by the first AI estimation means and image data including the vehicle outer shape, and estimates the vehicle identification code.

[0016] Therefore, not only can the vehicle identification code be estimated from the vehicle identification code area in each frame image data, but the vehicle outer shape area and vehicle identification code area in each frame image data can also be output, which enables various calculation processes to be performed using these vehicle outer shape areas and vehicle identification code areas.

[0017] (8) A vehicle identification code determination device according to the present invention is a vehicle identification code determination device that, when given frame image data in which still images including a vehicle's outer shape area and / or vehicle identification code area are arranged in chronological order, estimates a character string in the vehicle identification code area of ​​a certain vehicle from the frame image data, and includes: B) a first AI estimation means that, when given the frame image data as input data, estimates the vehicle outer shape area and the vehicle identification code area in that vehicle outer shape area in each frame image data, and outputs them as vehicle outer shape area identification data and vehicle identification code area identification data, respectively; and C) a second AI estimation means that extracts an image of the vehicle identification code area from the vehicle identification code area identification data estimated by the first AI estimation means and image data including the vehicle outer shape, and estimates the vehicle identification code.

[0018] Therefore, not only can the vehicle identification code be estimated from the vehicle identification code area in each frame image data, but the vehicle outline area and the vehicle identification code area in each frame image data can also be output, making it possible to perform various calculation processes using these vehicle outline areas and the vehicle identification code area.

[0019] (9) The vehicle identification code determination method of the present invention is a vehicle identification code determination method in which, when frame image data in which still images including the vehicle's outer shape area and / or vehicle identification code area are arranged in chronological order are given, a computer estimates the character string in the vehicle identification code area of ​​a certain vehicle from the frame image data, and includes the following steps. a first AI estimation step of estimating a vehicle exterior area and a vehicle identification code area in each frame image data when the frame image data is given as input data and outputting the area as vehicle exterior area identification data and vehicle identification code area identification data, respectively; a second AI estimation step of extracting an image of the vehicle identification code area from the vehicle identification code area identification data estimated in the first AI estimation step and image data including the vehicle exterior, and estimating the vehicle identification code; a tracking step of outputting 1) vehicle exterior area tracking result data and 2) vehicle identification code area tracking result data based on the vehicle exterior area estimated in the first AI estimation step and the image data thereof; 1) vehicle exterior area tracking result data generated by subjecting images in the vehicle exterior area to object tracking processing between previous and next frames of the time-series still image data, and 2) object tracking result data of images in the vehicle identification code area. vehicle specific code area tracking result data generated by object tracking of images in vehicle specific code areas performing tracking processing; an area linking step of linking the vehicle specific code area estimated in the first AI estimation step and the vehicle specific code area in each frame based on a predetermined decision rule and outputting the linked data as intra-frame linking result data; a step of assigning a common vehicle temporary identifier to the vehicle specific code area and the vehicle specific code area when the vehicle specific code area is linked to the vehicle specific code area in the intra-frame linking result data of a specific frame of the time-series still image data, wherein when tracking of the vehicle specific code area or the vehicle specific code area of ​​the specific frame has been achieved in at least one of the vehicle external area tracking result data or the vehicle specific code area tracking result data in a frame other than the specific frame, the assigned vehicle temporary identifier is:a temporary identifier assigning step of assigning the temporary vehicle identifier to the vehicle outer shape area or the vehicle identification code area of ​​the specified frame; a step of determining a vehicle identification code by associating the vehicle identification code estimated in each frame in the second AI estimation step with the vehicle identification code area to which the temporary vehicle identifier is assigned, wherein, when the vehicle identification codes assigned with the same temporary vehicle identifier in a plurality of frames do not match, adjustment is performed using a preset adjustment rule;

[0020] (10) The vehicle identification code determination method of the present invention is a vehicle identification code determination method in which, when frame image data in which still images including the vehicle's outer shape area and / or vehicle identification code area are arranged in chronological order are given, a computer estimates the character string in the vehicle identification code area of ​​a certain vehicle from the frame image data, and includes the following steps.a first AI estimation step of estimating a vehicle exterior area in each frame image data when the frame image data is given as input data and estimating a vehicle identification code area in that vehicle exterior area to output the vehicle exterior area identification data and vehicle identification code area identification data, respectively; a second AI estimation step of extracting an image of the vehicle identification code area from the vehicle identification code area identification data estimated in the first AI estimation step and image data including the vehicle exterior, and estimating the vehicle identification code; a tracking step of outputting 1) vehicle exterior area tracking result data and 2) vehicle identification code area tracking result data based on the vehicle exterior area estimated in the first AI estimation step and the image data thereof; 1) vehicle exterior area tracking result data generated by subjecting an image in the vehicle exterior area to object tracking processing between previous and next frames of the time-series still image data, and 2) vehicle identification code area generated by subjecting an image in the vehicle identification code area to object tracking processing, a step of assigning a vehicle temporary identifier common to the vehicle identification code area and the vehicle identification code area when the vehicle identification code area exists within the vehicle identification code area in the area tracking result data and a specific frame of the time-series still image data, and when tracking of the vehicle exterior area or the vehicle identification code area of ​​the specific frame is possible in at least one of the vehicle exterior area tracking result data or the vehicle identification code area tracking result data in a frame other than the specific frame, a temporary identifier assigning step of assigning the vehicle temporary identifier assigned to the vehicle exterior area or the vehicle identification code area of ​​the specific frame as the vehicle temporary identifier to be assigned; a step of determining a vehicle identification code by correlating the vehicle identification code estimated in each frame in the second AI estimating step with the vehicle identification code area to which the vehicle temporary identifier is assigned, and when vehicle identification codes assigned with the same vehicle temporary identifier do not match in a plurality of frames, a determination step of adjusting the vehicle identification code using a preset adjustment rule.

[0021] In this specification, the term "vehicle" is a concept that includes vehicles other than four-wheeled vehicles, such as two-wheeled vehicles. Furthermore, "vehicle body" refers to the "vehicle exterior." In the embodiments, the "vehicle identification code" corresponds to the vehicle number on the license plate. Note that in the embodiments, the vehicle number has been described as being composed of an "area name," "classification number," "hiragana," and "series designated number," but this is not limited to this and any unique ID that identifies the vehicle may be used. Furthermore, in the embodiments, the "vehicle exterior area" and the "vehicle identification code area" correspond to the "vehicle body frame" and the "license plate frame," respectively. Furthermore, in the embodiments, the "temporary vehicle identifier" corresponds to the "temporary vehicle ID." [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a diagram showing the overall configuration of a vehicle identification system 1 (first embodiment). [Figure 2] 1 is a diagram illustrating a hardware configuration of a vehicle identification system 1. FIG. [Figure 3] FIG. 2 is a diagram for explaining the relationship between steps in the first embodiment. [Figure 4] This is an example of time-series image data (frames f11 to f13). [Figure 5] This is an example of time-series image data (frames f14 to f15). [Figure 6] 10 is a diagram showing an example of the data structure of frame data estimated by the first and second estimation programs. [Figure 7] 10 is an example of data resulting from linking frame data. [Figure 8] This is an example of a case where there are multiple license plate frames within one vehicle frame. [Figure 9] 1 is an overall flowchart. [Figure 10] FIG. 11 is a detailed flowchart of steps S7 and S17. [Figure 11] This is an example of a case where the proportion of the license plate frame decreases. [Figure 12] This is an example of an incorrect license plate frame being detected. [Figure 13] FIG. 11 is a detailed flowchart of step S9. [Figure 14] 10 is a correspondence list of the temporary vehicle ID assignment and number reading process in FIG. 9. [Figure 15] FIG. 11 is a detailed flowchart of step S19. [Figure 16] 10 shows an example of a vehicle body ID, a vehicle body ID, and a license plate frame ID generated in step S7 and step S19. [Figure 17] FIG. 11 is a detailed flowchart of step S21. [Figure 18] FIG. 10 is a diagram illustrating tracking and linking in each frame. [Figure 19] FIG. 1 is a functional block diagram of the AI ​​estimation device 100 during learning. [Figure 20] The hardware configuration is shown in Figure 19. [Figure 21] FIG. 9 is a functional block diagram of a vehicle identification code determination device 900 (second embodiment). [Figure 22] FIG. 10 is a functional block diagram of an AI estimation device 970 during learning in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0024] (1. Overall Structure) FIG. 1 shows the configuration of a vehicle identification system 1 according to the present invention.

[0025] The vehicle identification system 1 includes an imaging data output means 73, a vehicle identification code determination device 100, a search means 87, a target vehicle identification code storage means 88, and a communication means 89. The imaging data output means 83 stores video images of the captured vehicle and outputs an image of each frame of the video. Each frame of the image contains vehicle exterior regions of zero, one, or two or more vehicles as vehicle image data. Furthermore, the vehicle exterior region may or may not contain the vehicle identification code region of the vehicle.

[0026] As will be described later, when an image of each frame of a moving image is given to the vehicle identification code determination device 100, the vehicle identification code determination device 100 outputs a vehicle image and a vehicle identification code.

[0027] The target vehicle identification code storage means 88 receives and stores the vehicle identification code of the target vehicle to be searched for from a server (not shown) via the communication means 89. The search means 87 determines whether the vehicle identification code determined by the vehicle identification code determination device 100 matches the vehicle identification code of the target vehicle stored in the target vehicle identification code storage means 78, and if they match, notifies the operator.

[0028] The following describes the vehicle identification code determination device 100. The vehicle identification code determination device 100 includes an AI estimation device 70, an area linking means 76, a tracking means 75, a temporary identifier assignment means 71, and a determination means 78.

[0029] The AI ​​estimation device 70 has a first estimation means 71 and a second estimation means 72. When the frame image data is given as input data, the first estimation means 71 estimates a vehicle exterior area and a vehicle specific code area in each frame image data, and outputs the vehicle exterior area identification data and vehicle specific code area identification data, respectively.

[0030] The second estimation means 72 extracts an image of the vehicle identification code area from the vehicle identification code area identification data estimated by the first AI estimation means 71 and image data including the vehicle's external shape, and estimates the vehicle identification code.

[0031] The tracking means 75 outputs, based on the vehicle exterior area and its image data estimated by the first AI estimation means 71, d1) vehicle exterior area tracking result data generated by performing object tracking processing of images in the vehicle exterior area between previous and next frames of the time-series still image data, and d2) vehicle identification code area tracking result data generated by object tracking of images in the vehicle identification code area by performing object tracking processing of images in the vehicle identification code area.

[0032] The area linking means 76 links the vehicle specific code area estimated by the first AI estimation means 71 in each frame with the vehicle specific code area based on a predetermined decision rule, and outputs the result as intra-frame linking result data.

[0033] The temporary identifier assignment means 77 is a means for assigning a common vehicle temporary identifier to the vehicle specific code area and vehicle specific code area when the vehicle specific code area and vehicle specific code area are linked in the intra-frame linking result data of a specific frame, and when tracking of the vehicle specific code area or the vehicle specific code area of ​​the specific frame is possible in at least one of the vehicle external area tracking result data or the vehicle specific code area tracking result data in a frame other than the specific frame, the temporary identifier assignment means assigns the vehicle temporary identifier assigned to the vehicle external area or the vehicle specific code area of ​​the specific frame as the vehicle temporary identifier to be assigned.

[0034] The determination means 78 is a means for determining a vehicle identification code by associating the vehicle identification code estimated in each frame by the second AI estimation means 72 with the vehicle identification code area to which the vehicle temporary identifier is assigned, and if the vehicle identification codes assigned to the same vehicle temporary identifier in multiple frames do not match, adjustment is made using a preset adjustment rule.

[0035] Therefore, even if the vehicle identification codes determined between multiple frames do not match, adjustment processing is performed.

[0036] (2. Hardware Configuration of Vehicle Identification System 1) The vehicle identification system 1 includes a CPU 123, a memory 127, a data storage unit 126, an input device 28, an optical drive 25, a monitor 30, the optical drive 25, and a bus line 29. The CPU 23 controls each unit via the bus line 129 in accordance with each program stored in the data storage unit 126.

[0037] The data storage unit 126 includes an operating system program 26o (hereinafter abbreviated as OS), a main program 126p, a first estimation program 126f, and a second estimation program 126s.

[0038] The processing of the main program 126p, the first estimation program 126f, and the second estimation program 126s will be described later.

[0039] The hardware configuration of the vehicle identification system 1 shown in Fig. 1 will be described with reference to Fig. 2. Fig. 2 shows an example of the hardware configuration of the vehicle identification system 1 configured using a CPU.

[0040] The imaging device 300 includes a control box 300a and an imaging unit 300b. The control box 300a and the imaging unit 300b are connected by an SDI cable 29.

[0041] The control box 300a includes a CPU 123, a data storage unit 126, and a communication unit 131. The data storage unit 126 stores an operating system program 126o (hereinafter abbreviated as OS) and a main program 126p. It also includes a memory 127 for storing calculation results. The CPU 123 performs various calculation processes in accordance with the programs stored in the data storage unit 126.

[0042] The camera 127 has a shutter speed of 1 ms and captures image data at 30 FPS. For this reason, a VD timing signal (synchronization signal) is sent to the light emission control unit 125 every 1 / 30 seconds. The irradiation unit 128 emits light with a wavelength of 850 nm (near-infrared light) in accordance with instructions from the light emission control unit 125. This allows an image of the traveling vehicle to be captured.

[0043] The main program 126p stores image data received from the camera 127 in the memory 127. The communication unit 131 receives the license plate number of a vehicle to be searched (for example, a stolen vehicle) from a management server (not shown) and stores it in the memory 127.

[0044] The data storage unit 126d stores a first estimation program 126f and a second estimation program 126s in addition to a main program 126p.

[0045] As described below, the first estimation program 126f extracts the existing vehicle body frame and license plate frame from the given frame data. The second estimation program 126s determines the vehicle number using the image of the license plate frame. This makes it possible to determine the vehicle number of a vehicle in the image data captured by the camera 127. The main program 126p determines whether the vehicle number determined by the first estimation program 126f and the second estimation program 126s matches the vehicle number of the vehicle being searched for. If a matching vehicle number is found, the notification unit 132 notifies the operator.

[0046] In this way, the first estimation program 126f and the second estimation program 126s can determine the vehicle number of a vehicle in the image data captured by the camera 127. At that time, learning processing of two areas is performed taking into account the relationship between the vehicle body frame and the license plate frame, so the license plate frame can be detected more accurately, and the vehicle number can be determined from the license plate frame.

[0047] (3. Relationship of steps in each program) An overview of data processing by the first estimation program 126f, the second estimation program 126s, and the main program 126p will be described with reference to FIG.

[0048] In Figure 3, the first AI estimation step SF1 is processed by the first estimation program 126f, the second AI estimation step SF2 is processed by the second estimation program 126S, and the frame linking step SF3, tracking step SF4, vehicle ID assignment step SF5, and determination step SF6 correspond to processing by the main program 126p.

[0049] Below, an overview of the processing of the first estimation step SF1, the second estimation step SF2, the frame linking step SF3, and the tracking step SF4 when the five frame data of FIGS. 4 and 5 are given will be described.

[0050] (3.1 First estimation step SF1) In step SF1, a process of extracting a vehicle body frame that specifies a vehicle body frame and a license plate frame that specifies a license plate frame is performed by the first estimation program 126f shown in FIG.

[0051] The first estimation program 126f performs AI estimation using an image of a vehicle body including a license plate frame as input data and the vehicle body frame and the license plate frame (both including position information) as output data.

[0052] The first estimation program 126f is trained to use various images as input images, such as an image including a vehicle body frame and a license plate frame, an image of only a vehicle body frame, an image of only a license plate frame, and an image including neither a vehicle body frame nor a license plate frame, and to calculate the probability (likelihood) of the vehicle body frame and license plate frame (both including position information) as output data, and to output the vehicle body frame or license plate frame if the probability exceeds a threshold. In this embodiment, the likelihood is also output.

[0053] As a result, for example, in frame f11 of FIG. 4A, as shown in FIG. 6A, four frames, namely, vehicle body frames 141a11, 141b11, license plate frames 142a11, 142b11, and their position information are output. Although not shown, the likelihood of each vehicle body frame or license plate frame is also output. The reason for outputting such likelihoods is that while vehicle body frames are rarely subject to errors, license plate frames are small in size, and furthermore, for example, character strings written on the vehicle body are more likely to be misrecognized than vehicle body frames. The use of such likelihoods will be described later.

[0054] (3.2 Second estimation step SF2) In step SF2, the license plate is estimated using AI from the license plate frame and its image data identified in step SF1. This process is executed by the second estimation program 126s shown in Figure 2. The second estimation program 126s executes AI estimation using the license plate frame and its image data as input data and producing the license plate as output data.

[0055] The second estimation program 126S takes various images as input images, such as images including license plate frames and images without license plate frames, and outputs an estimation result such as the number "Shiga 501 Hi 5231" as shown in Figure 6F as output data.

[0056] In this embodiment, the image within the license plate frame is trained to be blurred and the symbol "?" indicating that the plate is unreadable is output. As a result, the second estimation program 126S outputs an estimation result in which the number is completely blurred, such as "??????????", or only partly blurred, such as "??????5278."

[0057] (3.3 Frame Linking Step SF3) In the frame linking step, a linking process is performed between the vehicle body frame that specifies the vehicle body frame and the license plate frame that specifies the license plate frame, as shown in Fig. 3. This linking process is performed for each frame.

[0058] For example, as already explained, in frame f11 of FIG. 4A, four frames are detected, namely, vehicle body frames 141a11, 141b11, and license plate frames 142a11, 142b11, as shown in FIG. 6A. Furthermore, the vehicle body frame ID and the license plate frame have their position information (coordinates) stored. Based on the positional relationship of these frames, a process of linking the two is performed. This process will be described later. As a result, as shown in FIG. 7A, group 1 is determined, in which vehicle body frame 141a11 (P11, P12) is linked to license plate frame 142a11 (Q11, Q12), and group 2 is determined, in which vehicle body frame 141b11 (P13, P14) is linked to license plate frame 142b11 (Q13, Q14).

[0059] Similarly, for Figure 4B, from the data of Figure 6B, group 1 is determined in which vehicle body frame 141a12 (P21, P22) and license plate frame 142a12 (Q21, Q22) are linked, and group 2 is determined in which vehicle body frame 141b12 (P23, P24) and license plate frame 142b12 (Q23, Q24) are linked, as shown in Figure 7B.

[0060] 4C, no license plate frames are extracted as shown in FIG. 6C. Therefore, as shown in FIG. 7C, Group 1, which includes only the vehicle body frame 141a13 (P31, P32), and Group 2, which includes only the vehicle body frame 141b13 (P33, P34), are determined.

[0061] 4D, as shown in FIG. 6D, two vehicle body frames are extracted, but only one license plate frame is extracted. In this case, as shown in FIG. 7D, group 1, which is linked to vehicle body frame 141b12 (P23, P24) and license plate frame 142b12 (Q23, Q24), is determined to be group 2, which is only the vehicle body frame 141b14 (P43, P44).

[0062] Similarly, in FIG. 4E, only the vehicle body frame 141a15 (P51, P52) and the license plate frame 142a15 (Q51, Q52) are linked.

[0063] The detailed process of this linking will be described later.

[0064] (3.4 Tracking Step SF4) In the tracking step, object tracking technology is used to track the vehicle body frame and license plate frame between the front and rear frames (step SF4). For example, in the case of FIG. 4A, as shown in FIG. 8A, four frames are detected: vehicle body frames 141a11, 141b11, and license plate frames 142a11, 142b11. In the case of FIG. 4B, as shown in FIG. 8B, four frames are detected: vehicle body frames 141a12, 141b12, and license plate frames 142a12, 142b12. In this embodiment, object tracking processing is performed on the vehicle body frame and the license plate frame. Specifically, the image of the vehicle body frame 141a11 is compared with the images of the vehicle body frames 141a12 and 141b12, and it is determined whether the object has moved to one of them or another.

[0065] For example, in the case of FIG. 8A to FIG. 8B, if vehicle body frame 141a11 moves to vehicle body frame 141a12 and vehicle body frame 141b11 moves to vehicle body frame 141b12, it is determined that license plate frame 142a11 moves to license plate frame 142a12 and license plate frame 142b11 moves to 142b12.

[0066] 8B to 8C, vehicle body frame 141a12 is tracked to have moved to vehicle body frame 141a13, vehicle body frame 141b12 is tracked to have moved to vehicle body frame 141b13, and license plate frame 142a12 and license plate frame 142b12 are tracked to have been lost (disappeared).

[0067] 8C to 8D, it is determined that the vehicle body frame 141a13 has moved to the vehicle body frame 141a14, the vehicle body frame 141b13 has moved to the vehicle body frame 141b14, and further that the license plate frame 142a14 has been newly detected.

[0068] 8D to 8E, it is determined that vehicle body frame 141a14 has moved to vehicle body frame 141a15, vehicle body frame 141b14 has disappeared, and license plate frame 142a14 has moved to license plate frame 142a15.

[0069] As object tracking technology, Simple Online and Realtime Tracking (SORT), ByteTrack, etc. may be adopted, but are not limited to these.

[0070] The vehicle ID assignment step SF5 and the determination step SF6 in FIG. 3 will be described later with reference to a flowchart.

[0071] (4. Flowchart of number reading process) Details of the process of tracking the vehicle body frame and license plate frame between multiple frames to improve the accuracy of reading the number according to this embodiment will be described with reference to Fig. 9. The following description will be given taking as an example a case where the CPU 123 is given the time-series frame images of five frames shown in Figs. 4A-C and 5A-B based on the main program 126p (see Fig. 2).

[0072] (4.1 Frame inference processing for frame image f11) The CPU 123 (see FIG. 2) initializes the frame number n (step S1 in FIG. 9) and acquires the nth image data stored in the memory 127 (step S3 in FIG. 9). In this case, n=1, so the frame image f11 in FIG. 4A is acquired. The CPU 123 performs first and second AI estimation processing using the first estimation program 126f and the second estimation program 126s (step S5). In the first and second AI estimation processing, the vehicle body frame and license plate frame are extracted, and the license plate is estimated from the image of the license plate frame. As a result, the vehicle body frame and license plate frame are extracted, and the license plate is extracted from the image of the license plate frame, as shown in FIGS. 6A and 6F. As already explained, the likelihood of each is also output. The same applies hereinafter, but a description of the output of the likelihood will be omitted.

[0073] (4.2 Linking process for frame image f11) Based on the main program 126p (see FIG. 2), the CPU 123 links the vehicle body frame and the license plate frame in the n-th frame image (step S7 in FIG. 9). This linking process will be described with reference to FIG.

[0074] The CPU 123 extracts all vehicle body frames and license plate frames (Step S301 in FIG. 10). In this case, as shown in FIG. 6A, four frames are extracted: vehicle body frames 141a11, 141b11, and license plate frames 142a11, 142b11.

[0075] The CPU 123 determines whether there is a frame that can be linked based on the tracking result (step S302). In this case, since there is no tracking result for the frame f11, the CPU 123 determines whether there is a remaining frame (vehicle body frame or license plate frame) (step S304).

[0076] In this case, there are two vehicle frames and two license plate frames remaining, so the CPU 123 initializes the target vehicle frame number i (i=1) (step S305) and determines whether the target vehicle frame number i exceeds the number of remaining vehicle frame numbers (step S307). In this case, the number of remaining vehicle frame numbers does not exceed "2," so the CPU 123 selects the first vehicle frame, "vehicle frame 141a11."

[0077] The CPU 123 initializes the target license plate frame number j (step S311) and determines whether the target license plate frame number j exceeds the remaining number of license plate frames (step S313). In this case, the target license plate frame number "1" does not exceed the remaining number of license plate frames "2", so the CPU 123 performs an overlap determination between the first vehicle body frame and the first license plate frame (step S315). In this case, an overlap determination is performed between the vehicle body frame 141a11 and the license plate frame 142a11.

[0078] In this embodiment, the overlap determination condition is whether or not two conditions are satisfied: condition i) "located below the vehicle body" and condition ii) "80% of the number overlaps with the vehicle body." Regarding condition i), the license plate frame 142a11 is located below the vehicle body frame 141a11, so the answer is "YES." This determination is made based on the coordinate positions of both. Regarding condition ii) "80% of the number overlaps with the vehicle body," in this case, the answer is "YES" because there is 100% overlap. The significance of condition ii) will be described later.

[0079] CPU 123 determines whether the overlap determination condition is met based on the determination result of step S315 (step S317). In this case, all the conditions are met with "YES", so license plate frame 142b11 is set as a candidate number for vehicle body frame 141a11 (step S319).

[0080] The CPU 123 increments the target license plate frame number j (step S321) and determines whether the target license plate frame number j exceeds the number of remaining license plate frames (step S313). In this case, the target license plate frame number "2" does not exceed the number of remaining license plate frames "2", so the CPU 123 performs an overlap determination between the first vehicle body frame and the second license plate frame (step S315). In this case, an overlap determination is performed between the vehicle body frame 141a11 and the license plate frame 142b11. In this case, for condition i), the license plate frame 142b11 is located below the vehicle body frame 141a11, so the result is "NO". Furthermore, for condition ii) "80% of the number overlaps with the vehicle body", the result is "NO" because there is no overlap at all.

[0081] The CPU 123 determines whether the conditions for determining duplication are met (step S317). In this case, since not all of the answers are "YES," the CPU 123 increments the target license plate frame number j (step S321) and determines whether the target license plate frame number "3" exceeds the remaining number of license plate frames, "2" (step S313). In this case, since the target license plate frame number exceeds the remaining number of license plate frames, the CPU 123 determines whether there is one or more candidates for the i-th vehicle body frame (step S323). In this case, since there is only one candidate license plate frame for the i-th vehicle body frame, the CPU 123 selects the license plate with the highest reliability (highest probability of being the number) from the candidate numbers for the registered vehicle bodies (step S325).

[0082] In this embodiment, the reliability is determined from the likelihood output by the first estimation program 126f. In this case, the license plate frame 142a11 is the only license plate frame associated with the vehicle body frame 141a11, so the license plate frame 142a11 is selected.

[0083] The CPU 123 links the i-th body frame with the selected license plate frame (step S327). In this case, the license plate frame 142a11 and the body frame 141a11 are linked. The CPU 123 adds the i-th body frame to the linking table (see FIG. 7A) (step S329). As a result, the group of the body frame 141a11 and the license plate frame is stored.

[0084] The CPU 123 increments the target vehicle frame number i (step S331) and determines whether the target vehicle frame number i exceeds the number of remaining vehicle frames (step S307). In this case, i=2, and the number of remaining vehicle frames is not exceeded, so the i-th vehicle frame, "vehicle frame 141b11," is selected. The CPU 123 initializes the target license plate frame number j (j=1) (step S311) and determines whether the target license plate frame number j exceeds the number of remaining license plate frames (step S313). In this case, the target license plate frame number "1" does not exceed the number of remaining license plate frames, so the CPU 123 determines whether the i-th vehicle frame and the j-th license plate frame overlap (step S315). In this case, an overlap determination is performed between the vehicle frame 141b11 and the license plate frame 142a11. In this case, condition i) is judged as "NO" because license plate frame 142a11 is not located below vehicle body frame 141a11. Also, condition ii) "80% of the number overlaps with the vehicle body" is judged as "NO" because there is no 100% overlap.

[0085] The CPU 123 determines whether the conditions for determining duplication are met (step S317). In this case, since all of the answers are not "YES," the CPU 123 increments the target license plate frame number j (step S321) and determines whether the target license plate frame number j exceeds the number of remaining license plate frames (step S313).

[0086] In this case, since the target license plate frame number j does not exceed the number of remaining license plate frames, the CPU 123 performs an overlap determination between the i-th vehicle body frame and the j-th license plate frame (step S315). Specifically, an overlap determination is performed between the vehicle body frame 141b11 and the license plate frame 142b11. In this case, both conditions i) and ii) are determined to be "YES". The CPU 123 determines whether the overlap determination conditions are met (step S317). In this case, since all are "YES", the license plate frame 142b11 is set as a candidate number for the vehicle body frame 141b11 (step S319).

[0087] The CPU 123 increments the target license plate frame number j (step S321) and determines whether the target license plate frame number j exceeds the number of remaining license plate frames (step S313). In this case, the target license plate frame number "3" exceeds the number of remaining license plate frames "2", so the CPU 123 determines whether there are one or more candidates for the i-th vehicle body frame (step S323). In this case, the number of candidate license plate frames for the i-th vehicle body frame is one or more, so the CPU 123 selects the license plate with the highest reliability (highest probability of being the number) from the candidate numbers for the registered vehicle bodies (step S325). In this case, the only license plate frame linked to vehicle body frame 141b11 is license plate frame 142b11, so license plate frame 142b11 is selected.

[0088] The CPU 123 links the i-th body frame with the selected license plate frame (step S327). In this case, the license plate frame 142b11 and the body frame 141b11 are linked. The CPU 123 adds the i-th body frame to the linking table (see FIG. 7A) (step S329). As a result, the group of the body frame 141b11 and the license plate frame is stored.

[0089] In this way, the process of step S7 in FIG. 9 is performed for all the vehicle body frames and license plate frames in the frame.

[0090] (4.3 Significance of special judgment regarding license plate frames) Using FIG. 11, the significance of condition ii) "80% of the number overlaps with the vehicle body" in step S315 will be explained. For example, in the state of FIG. 11A, the area ratio of the overlapping range between license plate frame 142 and vehicle body frame 141 to license plate frame 142 is 100%. In contrast, as shown in FIG. 11B, when license plate frame 142 and vehicle body frame 141 are captured so that they partially extend beyond the image area, the area ratio is approximately 0.8. Furthermore, as shown in FIG. 11C, as the amount by which license plate frame 142 and vehicle body frame 141 partially extend beyond the image area increases, the area ratio becomes even smaller. In this way, when the area ratio is small, the vehicle number cannot be accurately estimated.

[0091] Therefore, in this embodiment, the license plate frame 142 is determined to extend beyond the frame image, and it is determined whether the ratio (area of ​​the intersection of the license plate frame 142 with the vehicle frame 141 / area of ​​the license plate) is 0.8 or more.

[0092] In this case, it is necessary to determine whether there is an overhanging area, but for the license plate frame 142, the aspect ratio of the license plate frame is set in advance, and it is determined whether the range of the license plate frame estimated by the first estimation program 126f is within that range. In Figure 11, a case where the license plate frame is overhanging in the horizontal direction has been described, but the same applies to a case where the license plate frame is overhanging in the vertical direction.

[0093] In this way, in this embodiment, even license plate frames that have been linked to a certain vehicle frame are not excluded from consideration before the judgment in step S315 is made. As a result, multiple license plate frames may be extracted as candidates for one vehicle frame. This problem does not arise because the processes in steps S323 to S327 narrow it down to one. This process enables more accurate linking even in cases where an area that is not a license plate frame is erroneously detected.

[0094] (4.4 False detection of license plate frames) Using Figure 12, we will explain cases where license plate frame erroneous detection occurs. One case is when the license plate frame of another vehicle is mixed in with the vehicle body frame. For example, as shown in Figure 12A, the license plate frame of vehicle body frame 141a is license plate frame 142a, but license plate frame 142b of another vehicle body frame is included within vehicle body frame 141a.

[0095] Even when there is no such mixture, there may be something resembling a license plate frame within the vehicle frame. For example, as shown in Figure 14B, in vehicle frame 141c, in addition to original license plate frame 142c, frame 143c exists.

[0096] (4.5 Temporary vehicle ID determination process for frame image f11) When the process of step S7 in Fig. 9 is completed, the CPU 123 assigns temporary vehicle IDs and determines numbers for all groups generated in step S7 (step S9 in Fig. 9). In this case, since there are two groups for the first frame (frame f11 in Fig. 7A), the process of Fig. 13 is executed for these two groups.

[0097] The CPU 123 determines whether a vehicle frame is present for one group (step S107). In this case, since a vehicle frame is present, it is determined that a license plate frame is present (step S108). In this case, since a license plate frame is present, it is determined that the tracking state is "NN". Such a tracking state "NN" will be explained using FIG. 14. The tracking state "NN" is assigned the process of "assigning a new identical tracking ID to the license plate and vehicle" and the output of the license plate is assigned the process of "returning the read number". In this case, the temporary vehicle ID "C1" is assigned to the vehicle frame 141a11 and the license plate frame 142a11 of group 1. In addition, the number of the temporary vehicle ID "C1" is the number read for the license plate frame 142a11 in step S5 of FIG. 9. 「??????????」 will be granted.

[0098] Similarly, the temporary vehicle ID "C2" is assigned to the vehicle body frame 141b11 and the license plate frame 142b11 of group 1, and the number "Shiga 501-hi 5231" is assigned to the temporary vehicle ID "C2."

[0099] As a result, the "temporary vehicle ID, vehicle body frame, license plate frame, and number" shown in FIG. 16A are determined.

[0100] In FIG. 13, if the group contains only the body frame, only the license plate frame, or neither, the temporary vehicle ID is determined in step S113, step S116, or step S117 of FIG. 13, respectively.

[0101] (4.6 Frame inference processing for frame image f12) Next, the CPU 123 acquires the (n+1)th image data stored in the memory 127 (step S11 in FIG. 9). In this case, since n=1, the frame image f12 in FIG. 4B is acquired. The CPU 123 performs first and second AI estimation processing using the first estimation program 126f and the second estimation program 126s (step S13). In the first and second AI estimation processing, the vehicle body frame and license plate frame are extracted, and the license plate number is estimated from the image of the license plate frame. This AI estimation processing has already been explained, so its explanation will be omitted. As a result, the vehicle body frame and license plate frame are extracted, and the license plate number is extracted from the image of the license plate frame, as shown in FIGS. 6B and 6G.

[0102] (4.7 Tracking process in frame images f11 and f12) Based on the main program 126p (see FIG. 2), the CPU 123 executes object tracking processing on the images of the vehicle body frame and license plate frame between two frames (step S15).

[0103] The object tracking process will now be described. The CPU 123 determines the vehicle body frame or license plate frame that is in a tracking relationship from the nth and n+1th images. In this case, in the frame images shown in FIGS. 4A and 4B, it is determined that the vehicle body frames 141a11 and 141a12, and the vehicle body frames 141b11 and 141b12, are in a tracking relationship. It is also determined that the license plate frames 142a11 and 142a12, and the license plate frames 142b11 and 142b12, are in a tracking relationship. In this embodiment, the vehicle body frame and the license plate frame are both determined between frames by extracting feature amounts from the images.

[0104] (4.8 Linking process for frame image f12) Next, the CPU 123 links the vehicle body frame and the license plate frame in the (n+1)th frame image (step S17 in FIG. 9). This linking process will be described with reference to FIG.

[0105] The CPU 123 extracts all vehicle body frames and license plate frames (Step S301 in FIG. 10). In this case, as shown in FIG. 6B, four frames are extracted: vehicle body frames 141a12, 141b12, and license plate frames 142a12, 142b12.

[0106] The CPU 123 determines whether there is a frame that can be linked based on the tracking result (step S302). In this case, for frame f12, in step S15 of FIG. 9, it is determined that the vehicle body frames 141a12 and 141b12 of frame f12 are in a tracking relationship with the vehicle body frames 141a11 and 141b11 of frame f11. Also, it is determined that the license plate frames 142a12 and 142b12 of frame f12 are in a tracking relationship with the license plate frames 142a11 and 142b11 of frame f11, respectively (see FIG. 8).

[0107] Therefore, the process proceeds to step S303, where the CPU 123 links all frames that can be linked based on the tracking results, adds the linked vehicle body frames and license plate frames to the linking table, and removes the linked frames from the processing target (step S303). Specifically, as already explained, the vehicle body frames 141a12 and 141b12 and the license plate frames 142a12 and 142b12 of frame f12 are determined to be in a tracking relationship with the vehicle body frames 141a11 and 141b11 and the license plate frames 142a11 and 142b11 of frame f11, respectively (see FIG. 8). Furthermore, in frame f11, the vehicle body frame 141a11 and the license plate frame 142a11 are linked, and the vehicle body frame 141b11 and the license plate frame 142b11 are linked.

[0108] Based on the tracking results and the linking results in the previous frame, the vehicle body frame 141a12 and license plate frame 142a12 in frame f12, and the vehicle body frame 141b12 and license plate frame 142b12 in frame f12 are linked. All linked vehicle body frames and license plate frames are added to the linking table, and linked vehicle body frames and license plate frames are removed from the processing target. As a result, the number of remaining vehicle body frames and remaining license plate frames in frame f12 becomes "0".

[0109] The CPU 123 determines whether there are any remaining frames (step S304). Specifically, it determines whether the number of remaining vehicle frames and the number of remaining license plate frames are both "0". In this case, since both are "0", the process ends.

[0110] As a result, the linking result shown in FIG. 7B is obtained.

[0111] In this example, the license plate frame associated with the vehicle frame 141b12 was determined by using the tracking result. However, if such tracking result cannot be used, for example, if the extraction of the license plate frame fails in frame f11, then in frame f12, the two license plate frames 142a12 are not determined as candidates in step S315 of FIG. 10 because they are not located below each other, and the license plate frame 142b12 is adopted. (4.9 Temporary vehicle ID determination process for frame images f11 and f12) The CPU 123 assigns a temporary vehicle ID and reads the license plate number for each group (step S19). Details of step S19 are shown in FIG.

[0112] Here, in steps S15 and S17, as already explained, between frames f11 and f12, the body frame 141a11 and the body frame 141a12 are in a tracking relationship, and the license plate frame 142a11 linked to such body frame 141a11 is determined to be in a tracking relationship with the license plate frame 142a12 in frame f11.

[0113] Since there is a license plate frame 142a11 linked to the vehicle body frame 141a11, the CPU 123 proceeds from step S201 to step S203 and S219, and determines whether the same vehicle body frame exists in both the nth frame and the n+1th frame (step S219). In this case, the tracking result shows that the vehicle body frames 141a11 and 141a12 are in a tracking relationship, so it determines whether there is a license plate frame in both the nth frame and the n+1th frame (step S221). In this case, since there is a license plate frame in both the nth frame and the n+1th frame, the vehicle body frames 141a11 and 141a12 are determined to be in the tracking state "TT" (step S223).

[0114] In the tracking state "TT", as shown in FIG. 14B, the temporary vehicle ID is "linking the vehicle body ID and license plate ID with the existing vehicle ID (using the same tracking ID)". Therefore, as shown in FIG. 16B, the same vehicle ID "C1" is assigned to the vehicle body frame 141a12 and the license plate frame 142a12. Also, in the tracking state "TT", the number is "returning the read number". In this case, as shown in FIG. 6G, the number "??????5278" is estimated for the license plate frame 142a12. Therefore, this number is assigned. As a result, as shown in FIG. 16B, the temporary vehicle ID "C1", the vehicle body frame "141a12", the license plate frame 142a12, and the number "??????5278" are determined.

[0115] This process is repeated for all groups. In this case, the same process is performed for group 2, and the temporary vehicle ID "C2", the body frame "141b12", the license plate frame 142b12, and the license plate number "Shiga 501-hi 5231" are determined, as shown in FIG. 16B.

[0116] (4.10 Number adjustment process for frame images f12 and f13) When the process of step S19 in FIG. 9 is completed, number adjustment is performed on the front and rear frames for all temporary vehicle IDs (step S21).

[0117] The significance of this step will be explained. For each license plate frame, the number can be read by the second estimation program 126s, and when assigning a temporary vehicle ID, the number read from each license plate frame is linked to the temporary vehicle ID rather than inheriting the previous number. Therefore, in Figure 16B, "??????5278" is read as the number for temporary vehicle ID "C1," and in Figure 16D, "Shiga 501te 5278" is read. As such, different numbers may be linked to the same temporary vehicle ID. Of course, it is also possible to use the previous result as is, such as the tracking status "TL." However, the determination of the previous frame is not necessarily correct. In this embodiment, the process of step S29 in Figure 9 is executed to improve the determination accuracy even if only slightly.

[0118] Details of step S29 are shown in Fig. 17. CPU 123 reads out the numbers of the n&n+1 frames (step S401), and determines the moving direction of the vehicle in the two frame images (step S402).

[0119] In this embodiment, the positional relationship between the vehicle body frame and the license plate frame is compared, and if the vehicle is moving downward in each frame image, it is determined that the vehicle is moving in the approaching direction, and if the vehicle is moving upward, it is determined that the vehicle is moving away. Also, in this embodiment, whether the vehicle is moving in the approaching direction is determined based on one of the vehicle bodies present in this frame. In this case, when comparing the vehicle body frames 141b11 and 141b12, it is determined that the vehicle is moving downward, so it is determined that the vehicle is moving in the approaching direction.

[0120] The CPU 123 determines whether all characters have been determined for the character groups "area name," "classification number," "hiragana," and "series designation number" (step S403). Here, the character groups "area name," "classification number," "hiragana," and "series designation number" correspond to, for example, the area name "Osaka," the classification number "555," the hiragana "to," and the series designation number "1234."

[0121] In this case, since all characters have not been determined, the CPU 123 selects an unprocessed character group in the n+1 frame (step S405). Here, the temporary vehicle ID selects the area name "??", which is the first character group.

[0122] The CPU 123 determines whether the movement direction is "approaching" or "moving away" in the nth and n+1th frames (step S407). In this case, it is the approaching direction, so the CPU 123 determines whether the character group in the n+1th frame is an unknown "?" (step S409). In this case, it is unknown, so the inference result in the n+1th frame is not adopted, and the inference result in frame n is adopted. In this case, the inference result in frame n is "??", so the area name is "??".

[0123] The CPU 123 determines whether all characters have been determined for the character groups "area name," "classification number," "hiragana," and "series designation number" (step S403). In this case, since all characters have not been determined, the CPU 123 selects an unprocessed character group in the n+1 frame (step S405). Here, the second character group, the classification number "???," is selected.

[0124] CPU 123 determines whether the movement direction is "approaching" or "moving away" in frames n and n+1 (step S407). In this case, it is the approaching direction, so CPU 123 determines whether the character group in frame n+1 is unknown "?" (step S409). In this case, it is unknown, so the inference result in frame n is not adopted, and the inference result in frame n is adopted. In this case, the inference result in frame n is "???", so the classification number is "???".

[0125] The CPU 123 determines whether all characters have been determined for the character groups "area name," "classification number," "hiragana," and "series designation number" (step S403). In this case, since all characters have not been determined, the CPU 123 selects an unprocessed character group in the n+1 frame (step S405). Here, the third character group, the hiragana "?", is selected.

[0126] The CPU 123 determines whether the movement direction is "approaching" or "moving away" in the nth and n+1th frames (step S407). In this case, it is the approaching direction, so the CPU 123 determines whether the character group in the n+1th frame is an unknown "?" (step S409). In this case, it is unknown, so the inference result in the n+1th frame is not adopted, and the inference result in frame n is adopted. In this case, the inference result in frame n is "?", so it becomes the hiragana "?".

[0127] The CPU 123 determines whether all characters have been determined for the character groups "area name," "classification number," "hiragana," and "series designation number" (step S403). In this case, since all characters have not been determined, the CPU 123 selects an unprocessed character group in the n+1 frame (step S405). Here, the fourth character group, the series designation number "5278," is selected.

[0128] The CPU 123 determines whether the movement direction is "approaching" or "moving away" in the nth and n+1th frames (step S407). In this case, it is the approaching direction, so the CPU 123 determines whether the character group in the n+1th frame is an unknown "?" (step S409). In this case, it is not unknown, so the inference result "5278" in the n+1th frame is adopted (step S411).

[0129] As a result, the temporary vehicle ID "C1" becomes number "??????5278".

[0130] The CPU 123 determines whether all characters have been determined for the character groups "area name," "classification number," "hiragana," and "series designation number" (step S403). In this case, all characters have been determined, so the process of FIG. 17 ends.

[0131] The CPU 123 performs this process for all temporary vehicle IDs (step S21 in FIG. 8). In this case, a similar determination is made for the temporary vehicle ID "C2," and the inference result for the n+1 frame is adopted for all character groups in step S411. In this case, the temporary vehicle ID "C2" is determined to be the license plate number "Shiga 501-hi 5231."

[0132] CPU 123 determines whether there is next frame image data (step S23 in FIG. 9), and if there is, stores the position information of the vehicle body frame and license plate frame in the subsequent frame as the results of the previous frame to be used in the next tracking process (step S25). In this case, the vehicle body frame and license plate frame in FIG. 8B for frame f12, which is the subsequent frame, and the images in each frame are stored as the results of the next previous frame.

[0133] (4.11 Frame inference processing for frame image f13) When the above adjustment process is completed, CPU 123 increments frame number n and acquires the (n+1)th image data stored in memory 127 (step S11 in FIG. 9). In this case, n=2, so frame image f13 in FIG. 4C is acquired.

[0134] The CPU 123 performs first and second AI estimation processes on the frame image f13 using the first estimation program 126f and the second estimation program 126s (step S13). As a result, as already explained, the vehicle body frame and license plate frame are extracted, and the license plate is estimated from the image of the license plate frame. As a result, the vehicle body frame is extracted as shown in FIG. 6C. In this case, since there is no license plate frame, only the vehicle body frame is available as frame data, and as a result, the license plate is not extracted.

[0135] (4.12 Tracking process in frame images f12 and f13) Based on the main program 126p (see FIG. 2), the CPU 123 executes object tracking processing on the images of the vehicle body frame and license plate frame between two frames (step S15).

[0136] The CPU 123 determines the vehicle body frame or license plate frame that is in a tracking relationship from the nth and n+1th images. In this case, the CPU 123 determines that, in the frame images shown in Figures 4B and 4C, the vehicle body frames 141a12 and 141a13, and the vehicle body frames 141b12 and 141b13, are in a tracking relationship. Note that, because a license plate frame is not extracted from frame f13, the CPU 123 determines that there is no tracking relationship.

[0137] (4.13 Linking process for frame image f13) Next, the CPU 123 links the vehicle body frame and the license plate frame in the (n+1)th frame image (step S17 in FIG. 9).

[0138] Such linking processing will be described with reference to FIG.

[0139] The CPU 123 extracts all vehicle body frames and license plate frames (Step S301 in FIG. 10). In this case, as shown in FIG. 6C, two frames of vehicle body frames 141a13 and 141b13 are extracted.

[0140] The CPU 123 determines whether there is a frame that can be linked based on the tracking results (step S302). In this case, for frames f12 and f13, the body frame 141a13 of frame f13 is in a tracking relationship with the body frame 141a12 of frame f12, and the body frame 141b13 of frame f13 is in a tracking relationship with the body frame 141b12 of frame f12. However, there is no license plate frame that can be linked. Therefore, the CPU 123 determines that there is a remaining frame (body frame or license plate frame), initializes the target body frame number i (step S305), and determines whether the target body frame number i exceeds the number of remaining body frames (step S307). In this case, the number of remaining body frames does not exceed "2," so the i-th body frame, "body frame 141a13," is selected.

[0141] The CPU 123 initializes the target license plate frame number j (j=1) (step S311) and determines whether the target license plate frame number j exceeds the remaining number of license plate frames (step S313). In this case, the target license plate frame number is "0", so it is determined that the target license plate frame number j exceeds the remaining number of license plate frames, and the CPU 123 determines whether the number of candidates for the i-th vehicle body frame is 1 or more (step S323). In this case, the candidate license plate frame for the i-th vehicle body frame is "0", so the CPU 123 adds the i-th vehicle body frame to the linking table (see FIG. 7C) (step S329). As a result, the vehicle body frame 141a13 is stored alone in the linking table.

[0142] The CPU 123 increments the target vehicle frame number i (step S331) and determines whether the target vehicle frame number i has exceeded the number of remaining vehicle frame slots (step S307). In this case, i=2, which does not exceed the number of remaining vehicle frame slots of "2," so the second vehicle frame "vehicle frame 141b13" is selected. The CPU 123 initializes the target license plate frame number j (j=1) (step S311) and determines whether the target license plate frame number j has exceeded the number of remaining license plate slots (step S313). In this case, the number of target license plate slots is "0," so it is determined that the target license plate frame number j has exceeded the number of remaining license plate slots, and the CPU 123 determines whether the number of candidates for the i-th vehicle frame is 1 or more (step S323). In this case, the candidate license plate frame of the i-th body frame is "0", so the CPU 123 adds the i-th body frame to the linking table (see FIG. 7C) (step S329). As a result, the body frame 141b13 is stored alone in the linking table.

[0143] The CPU 123 increments the target vehicle frame number i (step S331) and determines whether the target vehicle frame number i exceeds the number of remaining vehicle frames (step S307). In this case, since the number of remaining vehicle frames exceeds "2," if there are any license plate frames that are not linked to a vehicle frame, they are all added individually to the linking table (step S341). In this case, there are no license plate frames that are not linked to a vehicle frame, so the process ends. This results in a linking result consisting only of vehicle frames, as shown in FIG. 7C.

[0144] (4.14 Temporary vehicle ID determination process for frame images f12 and f13) The CPU 123 assigns a temporary vehicle ID and reads the license plate number based on the tracking result in step S15 and the linking result in step S17 (step S19 in FIG. 9).

[0145] Here, in steps S15 and S17, as already explained, between frames f12 and f13, it is determined that body frame 141a12 is in a tracking relationship with body frame 141a13, and there is no license plate frame linked to body frame 141a12.

[0146] Therefore, for the vehicle body frame 141a12, the CPU 123 proceeds from step S201 to step S203 in FIG. 15 and determines whether or not there is a license plate frame in the nth frame (step S203). Since there is a license plate frame in the nth frame, it determines whether or not there is an identical vehicle body frame in the nth and n+1th frames (step S219). In this case, since the vehicle body frame 141a12 is in a tracking relationship with the vehicle body frame 141a13, it is determined that there is an identical vehicle body frame, and it is determined whether or not there is a license plate frame in both nth and n+1th frames (step S221). In this case, there is not a single license plate frame in nth frame, and naturally there is no corresponding license plate frame, so it is determined that the vehicle body frames 141a12 and 141a13 are in the tracking state "TL" (step S225).

[0147] As shown in FIG. 14C, in the tracking state "TL", the temporary vehicle ID is "return the number linked to the vehicle body ID". Therefore, the vehicle ID "C1", which is the same as the vehicle body frame 141a12 shown in FIG. 16B, is assigned to the vehicle body frame 141a13. Also, in the tracking state "TL", the number is "return the read number". In this case, the number "??????5278" assigned to the temporary vehicle ID is returned. As a result, as shown in FIG. 16C, the temporary vehicle ID "C1", vehicle body frame "141a13", license plate frame "-", and number "??????5278" are determined.

[0148] This process is repeated for all groups. In this case, the same process is performed for group 2, and the tracking status is determined to be "TL" in step S225 of FIG. 15. Therefore, as shown in FIG. 16B, the temporary vehicle ID "C2", body frame "141b13", license plate frame "-", and license plate number "Shiga 501-hi 5231" are determined.

[0149] (4.15 Number adjustment process for frame images f12 and f13) When the process of step S19 in FIG. 9 is completed, the CPU 123 adjusts the number plates in the front and rear frames for all temporary vehicle IDs (step S21 in FIG. 9). In this case, the read numbers for the temporary vehicle IDs "C1" and "C2" in FIGS. 16B and 16C are the same. Therefore, the result will not change even if the process shown in FIG. 17 is performed.

[0150] The CPU 123 determines whether or not there is next frame image data (step S23 in FIG. 9), and if there is next frame image data, stores the position information of the vehicle body frame and license plate frame in the subsequent frame as the results of the previous frame (step S25).

[0151] (4.16 Frame inference processing for frame image f14) The CPU 123 increments the frame number n and acquires the (n+1)th image data stored in the memory 127 (step S11 in FIG. 9). In this case, n=3, so the frame image f14 in FIG. 5A is acquired.

[0152] The CPU 123 performs first and second AI estimation processing on the frame image f14 using the first estimation program 126f and the second estimation program 126s (step S13). As a result, as already explained, the vehicle body frame and license plate frame are extracted, and the license plate is estimated from the image of the license plate frame. As a result, the vehicle body frame and license plate frame are extracted as shown in FIG. 6D. The license plate is also extracted from the license plate frame.

[0153] (4.17 Tracking process in frame images f13 and f14) Based on the main program 126p (see FIG. 2), the CPU 123 executes object tracking processing on the images of the vehicle body frame and license plate frame between two frames (step S15).

[0154] The CPU 123 determines the vehicle body frame or license plate frame that is in a tracking relationship from the nth and n+1th images. In this case, with regard to the frame images shown in Figures 4C and 5A, it is determined that the vehicle body frames 141a13 and 141a14, and the vehicle body frames 141b13 and 141b14, are in a tracking relationship. In this case, since the license plate frame is not extracted in frame f13, it is determined that there is no tracking relationship.

[0155] (4.18 Linking process in frame image f14) Next, the CPU 123 links the vehicle body frame and the license plate frame in the (n+1)th frame image (step S17 in FIG. 9).

[0156] Such linking processing will be described with reference to FIG.

[0157] The CPU 123 extracts all vehicle body frames and license plate frames (Step S301 in FIG. 10). In this case, as shown in FIG. 6D, three frames are extracted: vehicle body frames 141a14, 141b14, and license plate frame 142a14.

[0158] The CPU 123 determines whether there is a frame that can be linked based on the tracking result (step S302). In this case, the vehicle body frame 141a14 of frame f14 is in a tracking relationship with the vehicle body frame 141a13 of frame f13, and the vehicle body frame 141b14 of frame f14 is in a tracking relationship with the vehicle body frame 141b13 of frame f13, but the license plate frame 142a14 of frame f14 does not exist in frame f13. Therefore, since there is no vehicle body frame or license plate frame that can be linked based on the tracking result, the process proceeds to step S304, where the CPU 123 determines whether there is a remaining frame (vehicle body frame or license plate frame) (step S304).

[0159] In this case, since there are remaining frames, the CPU 123 initializes the target vehicle frame number i (i=1) (step S305) and determines whether the target vehicle frame number i exceeds the number of remaining vehicle frame numbers (step S307). In this case, since the number of remaining vehicle frame numbers is not exceeded, which is "2", the first vehicle frame 141a14 is selected.

[0160] The CPU 123 initializes the target license plate frame number j (j=1) (step S311), and determines whether the target license plate frame number j exceeds the remaining number of license plate frames (step S313). In this case, the target license plate frame number is "1", so the CPU 123 determines that the target license plate frame number j does not exceed the remaining number of license plate frames, and performs an overlap determination between the first vehicle body frame and the first license plate frame (step S315). In this case, an overlap determination is performed between the vehicle body frame 141a14 and the license plate frame 142a14.

[0161] In this case, the answer to condition i) is "YES" because license plate frame 142a11 is located below vehicle body frame 141a11. The answer to condition ii) is "80% of the number overlaps with the vehicle body" because 100% overlaps.

[0162] The CPU 123 determines whether the conditions for determining duplication are met (step S317). In this case, all are "YES," so the CPU 123 sets the license plate frame 142a14 as a candidate number for the vehicle body frame 141a14 (step S319).

[0163] The CPU 123 increments the target license plate frame number j (step S321) and determines whether the target license plate frame number j has exceeded the number of remaining license plate frames (step S313). In this case, since the target license plate frame number j has exceeded the number of remaining license plate frames, the CPU 123 determines whether the number of candidates for the i-th vehicle body frame is 1 or more (step S323). In this case, the candidate license plate frame for the i-th vehicle body frame is "1," so the CPU 123 selects the license plate with the highest reliability (highest probability of being the number) from the candidate numbers for the registered vehicle bodies (step S325). In this case, the only license plate frame linked to vehicle body frame 141a14 is license plate frame 142a14, so license plate frame 142a14 is selected.

[0164] The CPU 123 links the i-th body frame with the selected license plate frame (step S327). In this case, the license plate frame 142a14 and the body frame 141a14 are linked. The CPU 123 adds the i-th body frame to the linking table (see FIG. 7D) (step S329). As a result, group 1 of the body frame 141a14 and the license plate frame 142a14 is stored.

[0165] The CPU 123 increments the target vehicle frame number i (step S331) and determines whether the target vehicle frame number i exceeds the number of remaining vehicle frames (step S307). In this case, i=2, and the number of remaining vehicle frames is not exceeded, so the i-th vehicle frame, "vehicle frame 141b14," is selected. The CPU 123 initializes the target license plate frame number j (step S311) and determines whether the target license plate frame number j exceeds the number of remaining license plate frames (step S313). In this case, the number of target license plate frames is "1," so the CPU 123 determines that the target license plate frame number j does not exceed the number of remaining license plate frames, and performs an overlap determination between the i-th vehicle frame and the j-th license plate frame (step S315). In this case, an overlap determination is performed between the vehicle frame 141b11 and the license plate frame 142a14. Regarding condition i), the license plate frame 142a11 is not located below the vehicle body frame 141a11, so the answer is "NO." Regarding condition ii), "80% of the number overlaps with the vehicle body," the overlap is 0% in this case, so the answer is "NO."

[0166] The CPU 123 determines whether the conditions for overlap determination are met based on the determination result of step S315 (step S317). In this case, all are "NO" and the conditions are not met, so the CPU 123 increments the target license plate frame number j (step S321) and determines whether the target license plate frame number j exceeds the number of remaining license plate frames (step S313). In this case, the target license plate frame number j exceeds the number of remaining license plate frames, so the CPU 123 determines whether the number of candidates for the i-th vehicle body frame is 1 or more (step S323). In this case, the candidate license plate frame for the i-th vehicle body frame is "0", so the CPU 123 adds the i-th vehicle body frame to the linking table (see FIG. 7D) (step S329). As a result, the vehicle body frame 141b14 is stored independently.

[0167] The CPU 123 increments the target vehicle frame number i (step S331) and determines whether the target vehicle frame number i exceeds the number of remaining vehicle frames (step S307). In this case, since the number of remaining vehicle frames exceeds "2," if there are any license plate frames that are not linked to a vehicle frame, they are all added individually to the linking table (step S341). In this case, there are no license plate frames that are not linked to a vehicle frame, so the process ends. This results in the linking result shown in FIG. 7D.

[0168] (4.19 Temporary vehicle ID determination process for frame images f13 and f14) Based on the tracking result in step S15 and the linking result in step S17, CPU 123 assigns a temporary vehicle ID and reads the license plate (step S19 in FIG. 9). As already explained, in steps S15 and S17, between frames f13 and f14, body frame 141a13 is in a tracking relationship with body frame 141a14, and body frame 141a14 and license plate frame 142a14 are linked.

[0169] Therefore, for the vehicle body frame 141a13, the CPU 123 proceeds from step S201 to step S203 and determines whether or not there is a license plate frame in the nth frame (step S203). In this case, since there is no license plate frame in the nth frame, it determines whether or not there is an identical vehicle body frame in the nth and n+1th frames (step S205). In this case, since the vehicle body frame 141a13 is in a tracking relationship with the vehicle body frame 141a14, it is determined that there is an identical vehicle body frame, and it determines whether or not there is a license plate frame linked to the vehicle body frame 141a14 in the nth frame (step S207). In this case, since the license plate frame 142a14 is present in the nth frame, it is determined that the vehicle body frames 141a13 and 141a14 are in the tracking state "TN" (step S211).

[0170] As shown in FIG. 14B, in the tracking state "TN", the temporary vehicle ID is "link the existing temporary vehicle ID that is linked to the vehicle body ID with the license plate frame." Therefore, as shown in FIG. 16D, the vehicle ID "C1" assigned to the vehicle body frame 141a13 is assigned to the vehicle body frame 141a14. Also, in the tracking state "TN", the number is "return the read number." In this case, the number "Shiga 501 te 5278" estimated by the second estimation program 26s is returned. As a result, as shown in FIG. 16D, the temporary vehicle ID "C1", the vehicle body frame "141a14", the license plate frame "142a14", and the number "Shiga 501 te 5278" are determined.

[0171] This process is repeated for all groups. In this case, the same process is performed for group 2. In this case, in step S213 of FIG. 14, the tracking status is determined to be "TL" (step S213), so as shown in FIG. 16D, the temporary vehicle ID "C2", body frame "141b13", license plate frame "-", and license plate number "Shiga 501-hi 5231" are determined.

[0172] (4.20 Number adjustment process for frame images f13 and f14) When the processing of step S19 in FIG. 9 is completed, the CPU 123 adjusts the number plates in the front and rear frames for all temporary vehicle IDs (step S21). In this case, for temporary vehicle ID "C1," the number plates in FIGS. 16C and 16D are "??????5278" in the nth frame and "Shiga 501 te 5278" in the n+1th frame. Therefore, in step S411 in FIG. 17, "Shiga 501 te 5278" in the n+1th frame is adopted. Furthermore, for temporary vehicle ID "C2," the two are the same and are not unknown, so in step S411 the inference result in the n+1th frame is adopted, and the number plate becomes "Shiga 501 hi 5231."

[0173] The CPU 123 determines whether or not there is next frame image data (step S23 in FIG. 9), and if there is next frame image data, stores the position information of the vehicle body frame and license plate frame in the subsequent frame as the results of the previous frame (step S25).

[0174] (4.21 Frame inference processing for frame image f15) The CPU 123 increments the frame number n and acquires the (n+1)th image data stored in the memory 127 (step S11 in FIG. 9). In this case, since n=4, the frame image f15 in FIG. 5B is acquired.

[0175] The CPU 123 performs first and second AI estimation processing on the frame image f15 using the first estimation program 126f and the second estimation program 126s (step S13). As a result, as already explained, the vehicle body frame and license plate frame are extracted, and the license plate is estimated from the image of the license plate frame. As a result, the vehicle body frame and license plate frame are extracted as shown in FIG. 6E. The license plate is also extracted from the license plate frame.

[0176] (4.22 Tracking process in frame images f14 and f15) Based on the main program 126p (see FIG. 2), the CPU 123 executes object tracking processing on the images of the vehicle body frame and license plate frame between two frames (step S15).

[0177] In this case, in the frame images shown in FIGS. 4D and 4E, it is determined that the vehicle body frames 141a14 and 141a15, and the license plate frames 142a14 and 142a15 are in a tracking relationship.

[0178] (4.23 Linking process for frame image f15) Next, the CPU 123 links the vehicle body frame and the license plate frame in the (n+1)th frame image (step S17 in FIG. 9). This linking process will be described with reference to FIG.

[0179] The CPU 123 extracts all vehicle body frames and license plate frames (Step S301 in FIG. 10). In this case, as shown in FIG. 6E, three frames are extracted: vehicle body frames 141a15, 141b15, and license plate frame 142a15.

[0180] The CPU 123 determines whether there are any vehicle body frames or license plate frames that can be linked based on the tracking results (step S302). In this case, in step S15 of FIG. 9, the vehicle body frame 141a15 of frame f15 is in a tracking relationship with the vehicle body frame 141a14 of frame f14, and the license plate frame 142a15 of frame f15 is in a tracking relationship with the license plate frame 142a14 of frame f14. Also, the vehicle body frame 141a14 of frame f14 and the license plate frame 142a14 are already linked. Therefore, the process proceeds to step S303, where all frames that can be linked based on the tracking results are linked, and the linked vehicle body frames and license plate frames are added to the linking table, while the linked frames are removed from the processing target (step S303). As a result, the number of remaining vehicle body frames and the number of remaining license plate frames in frame f15 become "0."

[0181] The CPU 123 determines whether there are any remaining frames (step S304). Specifically, it determines whether the number of remaining vehicle frames and the number of remaining license plate frames are both "0". In this case, since both are "0", the process ends.

[0182] As a result, the linking result shown in FIG. 7E is obtained.

[0183] (4.24 Temporary vehicle ID determination process for frame images f14 and f15) Based on the tracking result in step S15 and the linking result in step S17, CPU 123 assigns a temporary vehicle ID and reads the license plate (step S19). As already described, between frames f14 and f15, body frame 141a14 is in a tracking relationship with body frame 141a15, and body frame 141a15 and license plate frame 142a15 are linked.

[0184] Therefore, the CPU 123 proceeds from step S201 to steps S203, S205, S207, and S211 in FIG. 15 for the body frame 141a14, and determines that the body frames 141a14 and 141a15 are in the tracking state "TT."

[0185] The CPU 123 determines the vehicle body frame or license plate frame that is in a tracking relationship from the nth and (n+1)th frame images. In this case, in the frame images shown in Figures 5A and 5B, it is determined that the vehicle body frames 141a14 and 141a15 are in a tracking relationship, and the license plate frame 142a15 linked to the vehicle body frame 141a15 that is in a tracking relationship with the vehicle body frame 141a12 is in a tracking relationship with the license plate frame 142a14.

[0186] In the tracking state "TT", as shown in FIG. 14B, the temporary vehicle ID is "linked to the vehicle body ID and license plate ID (using the same tracking ID)". Therefore, as shown in FIG. 16E, the same vehicle ID "C1" is assigned to the vehicle body frame 141a12 and the license plate frame 142a12. Also, in the tracking state "TN", the number is "return the read number". In this case, the number "Shiga 501 te 5278" estimated by the second estimation program 26s is returned. As a result, as shown in FIG. 16E, the temporary vehicle ID "C1", the vehicle body frame "141a15", the license plate frame "142a15", and the number "Shiga 501 te 5278" are determined.

[0187] This process is repeated for all groups. In this case, since only group 1 exists, the process of step S19 is terminated.

[0188] (4.25 Number adjustment process for frame images f14 and f15) When the process of step S19 in Fig. 9 is completed, the CPU 123 adjusts the number plates in the front and rear frames for all temporary vehicle IDs (step S21). In this case, the number plates in Figs. 16D and 16E are "Shiga 501 te 5278" in the nth frame and "Shiga 501 te 5278" in the n+1th frame. In other words, since both are the same, "Shiga 501 te 5278" in the n+1th frame is adopted in step S411 in Fig. 17.

[0189] The CPU 123 determines whether or not there is next frame image data (step S23). In this case, there is no next frame image data, so the process ends.

[0190] (5. Other) 9 is completed, the CPU 123 can use the number and its image thus read to search for whether the vehicle is a wanted vehicle by the main program 126p (see FIG. 2). The search process after the number is identified is the same as in the conventional case, so details will be omitted.

[0191] In this embodiment, when assigning the temporary vehicle ID in FIG. 16, the IDs of the frames extracted in each frame are directly used as the vehicle frame ID and license plate frame ID. However, this is not limited to this, and new IDs may be assigned. In this case, for example, in the case of the temporary vehicle ID "C1" in FIG. 16A, the temporary vehicle ID "C1," vehicle ID "B1," license plate ID "N1," and license plate number "????????????" are assigned in frame f11. Furthermore, since the tracking state is "TT" in frame f12, the vehicle ID "B1," license plate ID "N1," and license plate number "??????5278" are assigned in FIG. 16B. Then, after the license plate frame is temporarily lost in frame f13, the license plate frame is detected in frame f14. In this case, the tracking state in step S19 is "TN." Therefore, a new ID "N3" is assigned as the license plate frame ID. Even in this case, there is no particular problem because the vehicles are connected by the temporary vehicle ID. At this time, the same license plate ID may be assigned based on the temporary vehicle ID by referring to the previous license plate ID.

[0192] In this way, even if the vehicle ID and license plate ID are assigned IDs different from the vehicle frame and license plate frame in each frame, the number estimated by the second estimation program 26s can be read out by storing the correspondence between at least the license plate ID and the license plate frame in each frame.

[0193] (6. Regarding the pattern of temporary vehicle ID assignment) As shown in Figure 14A, there are a total of nine patterns for tracking the vehicle frame and license plate frame. This is because there are three possible outcomes for tracking each frame: "Tracking Successful," "Lost," and "New." These nine patterns are the tracking statuses "NL," "TN," "TT," "LN," "NT," "TL," "NN," "LT," and "LL." The assignment of temporary vehicle IDs and the output format of license plates differ depending on each of these patterns.

[0194] Some of the tracking states have been explained, but we will now briefly explain these nine patterns, including the remaining cases. In the following, to simplify the explanation, we will use as an example a case where one vehicle is captured in one frame image, but the same applies when there are multiple vehicles.

[0195] For example, if only the license plate frame is detected in the previous frame and the vehicle body frame is detected in the subsequent frame, the tracking state will be as follows.

[0196] Frame f1: The vehicle frame was newly detected, but the license plate frame could not be detected. Frame f2: The vehicle frame was tracked as is, and the license plate frame was newly detected. Frame f3: Both the body frame and the license plate frame were tracked.

[0197] In this case, the tracking states of frame f1, frames f1 and f2, and frames f2 and f3 are "NL," "TN," and "TT," respectively. The output of each temporary vehicle ID and license plate is as follows, as shown in FIG. 14B.

[0198] NL: Vehicle ID "Link new temporary vehicle ID", license plate output is "Number is undetermined", TN: Vehicle ID "Links the associated temporary vehicle ID with the license plate ID"; license plate output "Returns the read number"; TT: Temporary vehicle ID: "Continue to link vehicle ID & license plate ID with existing temporary vehicle ID", license plate output: "Returns the read number".

[0199] Also, if only the license plate frame is detected in the previous frame, and only the vehicle frame is detected in the subsequent frame, and then the license plate frame is lost, the tracking state will be as follows.

[0200] Frame f1: The vehicle frame was not detected, but the license plate frame was newly detected. Frame f2: The vehicle frame is newly detected, and the license plate is still tracked. Frame f3: The body frame was tracked, but the license plate frame was lost.

[0201] In this case, the tracking states of frame f1, frames f1 and f2, and frames f2 and f3 are "LN," "NT," and "TL," respectively. The output of each temporary vehicle ID and license plate is as follows, as shown in FIG. 14C.

[0202] LN: Temporary vehicle ID is "assign a new temporary vehicle ID to the license plate ID", and the license plate output is "return the read number". NT: Temporary vehicle ID is "assigning the temporary vehicle ID of the license plate to the vehicle", and license plate output is "returning the read number", TL: Temporary vehicle ID is "Continue tracking only the vehicle (using the same temporary vehicle ID)", and license plate output is "The license plate can be determined from the temporary vehicle ID assigned to the vehicle, so return that number", Also, if both the vehicle frame and the license plate frame are detected first, then the vehicle body is lost, and finally both are lost, the tracking state will be as follows. For example, this is the case below.

[0203] Frame f1: The vehicle frame and license plate frame were newly detected. Frame F2: Lost car frame, tracking license plate frame, Frame f3: Body frame lost, license plate frame lost, In this case, the tracking states are "NN", "LT", and "LL", respectively. The output of each temporary vehicle ID and license plate is as follows, as shown in Figure 14D.

[0204] NN: Temporary vehicle ID is "assigning a new temporary vehicle ID to the vehicle ID and license plate ID", and license plate output is "returning the read number", LT: Temporary vehicle ID is "Continue tracking only the license plate ID (use the same temporary vehicle ID)", license plate output is "Return the read number", LL: The temporary vehicle ID is "Lost" and the license plate output is "Lost".

[0205] In this way, by assigning a common temporary vehicle ID to the vehicle body ID or license plate ID, even if tracking using the vehicle body frame or license plate frame is interrupted along the way, if tracking is resumed using either the vehicle body frame or license plate frame, the vehicle can be recognized as the same vehicle.

[0206] Note that there are multiple combinations even for the same tracking state. This point will be explained using FIG. 15. For example, for tracking state "LN," there are cases where the vehicle body frame 141xn is present in the nth frame and not present in the n+1th frame, and cases where the vehicle body frame 141xn is not present in the nth frame and not present in the n+1th frame either. In these cases, the order is step S201 → step S203 → step S205 → step S209 → step S215, and step S201 → step S251 → step S253 → step S257 → step S265.

[0207] Similarly, for the tracking state "LT," there may be a case where the vehicle frame 141xn is present in the nth frame and not present in the n+1th frame, or a case where the vehicle frame 141xn is not present in the nth frame and not present in the n+1th frame either. In these cases, the order is step S201 → step S203 → step S219 → step S231 → step S233, or step S201 → step S251 → step S273 → step S277 → step S285.

[0208] Similarly, for the tracking state "NL," there may be a case where the license plate frame 142xn is not present in the nth frame and the license plate frame 142xn is present in the nth frame and the nth frame and the nth frame and the nth frame also do not have the vehicle body frame 141xn. In this case, the order is step S201 → step S251 → step S253 → step S255 → step S263, and step S201 → step S251 → step S273 → step S275 → step S283.

[0209] Furthermore, there are four patterns for the tracking state "LL." One is when the vehicle body frame 141xn is present in the nth frame and not present in the n+1th frame, and the other is when the vehicle body frame 141xn is not present in the nth frame and not present in the n+1th frame. In this case, the license plate frame may also be present in the nth frame and not present in the n+1th frame, or the other is when the vehicle body frame 141xn is not present in the nth frame and not present in the n+1th frame. Therefore, in this case, there are four cases: Step S201 → Step S203 → Step S205 → Step S209 → Step S217; Step S201 → Step S203 → Step S219 → Step S231 → Step S235; Step S201 → Step S251 → Step S253 → Step S257 → Step S267; and Step S201 → Step S251 → Step S273 → Step S277 → Step S287.

[0210] There is only one pattern for the tracking state "NN." This is when there is no vehicle body frame 141xn or license plate frame 142xn in the nth frame, and there is a vehicle body frame 141xn and a license plate frame 142xn in the n+1th frame. Therefore, the sequence is step S201 → step S251 → step S253 → step S255 → step S261.

[0211] In this embodiment, the temporary vehicle ID is used as a tracking ID that is commonly used for both vehicle tracking and license plate tracking. This allows the two to be linked using the tracking ID assigned to either one of the tracking IDs.

[0212] (7. Two types of frames across multiple frames, object tracking process for each, linking two frames in each frame, and the significance of assigning a temporary vehicle ID) Using FIG. 18, the object tracking process between multiple frames in this embodiment and the significance of linking the vehicle body frame and license plate frame in each frame and assigning a temporary vehicle ID will be described.

[0213] As already explained, when the five frame images of Figures 4 and 5 are given, the present system obtains the following tracking results between each frame (f11 to f15), as shown in Figure 18A, for the vehicle body frame: vehicle body frame 141a11 → 141a12 → 141a13 → 141a14 → 141a15, vehicle body frame 141b11 → 141b12 → 141b13 → 141b14, license plate frame 142a11 → 142a12, license plate frame 142b11 → 142b12, and license plate frame 142a14 → 142a15.

[0214] Also, as shown in FIG. 18B, the correspondences in each frame are as follows: in frame f11, vehicle body frame 141a11 and license plate frame 142a11, and vehicle body frame 141b11 and license plate frame 142b11; in frame f12, vehicle body frame 141a12 and license plate frame 141b12, and vehicle body frame 142a12 and license plate frame 142b12; in frame f13, only vehicle body frame 141a13 and only vehicle body frame 141b13; in frame f14, vehicle body frame 141a14 and license plate frame 142a14 and only vehicle body frame 141b14; and in frame f15, vehicle body frame 141a15 and license plate frame 142a15 are obtained.

[0215] From these two tables, a matrix of correspondences is obtained showing which vehicle frame and / or license plate frame each temporary vehicle ID is associated with, as shown in Figure 18C, and the second estimation program 26s associates the estimated number with this result. Therefore, as long as the vehicle can be tracked by either the vehicle frame or the license plate frame, it can be determined that it is the same vehicle.

[0216] Furthermore, the number estimated by the second estimation program 26s in each frame is adjusted between multiple frames. This allows for more accurate number determination. Furthermore, if the first estimation program 126f outputs two types of frames, the tracking process between multiple frames and the linking process in each frame can be performed in parallel, allowing for determination in a shorter time.

[0217] Based on these detection results, for example, by installing this system in a moving vehicle, it becomes possible to locate the target vehicle by its license plate number. It is also possible to install the system on the road or on the side of the road (for example, at an intersection).

[0218] (8. Learning of the AI ​​estimation device 70) We will now explain the learning process of the AI ​​estimation device 100 in Fig. 1. As already explained, the AI ​​estimation device 100 has two AI estimation means, and parameters are adjusted by the learning process described below.

[0219] 19, the AI ​​estimation device 100 includes a first AI estimating device 200 and a second AI estimating means 400. The first AI estimating device 200 includes a first adjustment means 202 and an unlearned first AI estimating means 201. The second AI estimating device 400 includes a second adjustment means 402 and an unlearned second AI estimating means 401.

[0220] The first AI estimation device 200 is provided with first training data for learning. In this embodiment, first supervised answer data and first example data (image data) are provided as the first training data. The first example data includes various images with and without vehicle body frames and / or license plate frames, and coordinates specifying the areas of the vehicle body frames and / or license plate frames are provided as supervised answer data. In this embodiment, both the vehicle body frame specifying data and the identifier area specifying data are rectangular data, but this is not limited to this.

[0221] When the first example data is given to the first AI estimating means 201, it outputs the estimated result data to the first adjusting means 202. Since the first correct answer data for the example data is given to the first adjusting means 202, the first adjusting means 202 adjusts the parameters of the first AI estimating means 201 so as to reduce the error between the estimated result data and the first correct answer data. In this way, the learning process of the first AI estimating means 201 is performed.

[0222] As already explained, the first AI estimation means 201 outputs rectangular data of the vehicle body frame and identifier region as estimation result data.

[0223] The second AI estimation device 400 is provided with rectangular data of the vehicle body frame and license plate frame output from the first AI estimation device 200. The image extraction means 403 receives the image data in addition to the estimation result data. Then, it provides license plate image data from the image data and the rectangular data of the license plate frame to the second AI estimation means 401. The second AI estimation means 401 estimates and outputs the license plate from the license plate image data.

[0224] The second adjustment means 402 is provided with second training data for learning. In this embodiment, second correct answer data and second example data (license plate image data) are provided as the second training data. The second accurate data is the vehicle's number on the license plate (hereinafter referred to as "number"). This number also includes cases where some or all of the character strings are unclear.

[0225] When the second example data is provided to the second AI estimation means 401, it outputs the estimated result data to the second adjustment means 402. Since the second adjustment means 402 is provided with the second correct answer data for the example data, it adjusts the parameters of the second AI estimation means 401 so as to reduce the error between the estimated result data and the second correct answer data. In this manner, the second AI estimation means 401 performs a learning process. Through this learning process, the second AI estimation means 401 estimates the character string within the license plate frame based on the rectangular data of the vehicle frame and the license plate frame and the image data used for the estimation. The first correct answer data and second correct answer data are provided by human judgment, as in general supervised learning.

[0226] (8.1 Hardware Configuration) The hardware configuration for realizing the AI ​​estimation device 100 will be described with reference to Fig. 20. The AI ​​estimation device 100 includes a CPU 23, memory 27, a hard disk 26, an input device 28, an optical drive 25, a monitor 30, the optical drive 25, and a bus line 29. The CPU 23 controls each component via the bus line 29 in accordance with each program stored on the hard disk 26.

[0227] The hard disk 26 has an operating system program 26o (hereinafter abbreviated as OS), a main program 26p, a first estimation program 26f, a second estimation program 26s, a frame data storage unit 26w, a number storage unit 26n, and an image data storage unit 26u.

[0228] As will be described later, the main program 26p controls the first estimating program 26f and the second estimating program 26s so that they adjust their parameters by repeatedly providing a plurality of image data.

[0229] The image data storage unit 26u stores multiple image data captured by a camera installed on the road, as shown in Figures 4A and 4B. The multiple image data include images that include the vehicle body outline and images that do not. Note that even in images that include the vehicle body outline, the first estimation program 26f may not be able to extract the vehicle body frame due to diffuse reflection, etc. Furthermore, such vehicle body outline may or may not include the license plate frame. The frame data storage unit 26w stores data that identifies the vehicle body frame and license plate frame estimated by the first estimation program 126f. The number storage unit 26n stores the license plate number estimated by the second estimation program 26s. Note that the first estimation program 26f and the second estimation program 26s are in an unlearned state.

[0230] In this embodiment, the rectangular data of the vehicle body frame and the license plate frame is specified by the coordinates of two points, but the invention is not limited to this.

[0231] The main program 26p provides this training data to the first estimation program 26f. The first estimation program 26f estimates the coordinates of the vehicle body frame 141 and the license plate frame 142 from the image data of the provided training data. Then, based on the estimation results and the provided correct answer data, the internal parameters are adjusted so as to reduce the error between them. This parameter adjustment is executed repeatedly.

[0232] (9. Embodiments with different AI estimation methods) In the above embodiment, the two regions are extracted independently without considering the relationship between the vehicle body frame 141 and the license plate frame 142. However, it is also possible to extract the two regions independently without considering the relationship between the two regions. This allows only the license plate frame related to the vehicle body frame to be extracted, making the area linking step in FIG. 3 unnecessary.

[0233] For example, image data such as that shown in FIG. 6B is provided as training data. In this training data, license plate frame 142c and area 143 exist within vehicle body frame 141c. Area 143 is a case where an area within the vehicle body frame is mistakenly recognized as a license plate frame. Using this training data, it is possible to learn that area 143 is not a license plate frame. Also, in FIG. 6B, license plate frame 142c exists, but training data in which license plate frame 142c exists outside the image is also learned. Using this training data, it is possible to learn that area 143 is not a license plate frame.

[0234] By providing such training data, it is possible to learn the areas of the vehicle body frame and the license plate frame, taking into account the relationship between the two.

[0235] A functional block diagram of this case is shown in Figure 21. The difference from Figure 1 is that there is no area linking means 76, and the first estimation means 971 outputs the vehicle body outer diameter area and the vehicle identification code area in a correlated state. The differences from the first embodiment will be explained below.

[0236] When frame image data is provided as input data, the first AI estimation means 971 estimates a vehicle exterior area in each frame image data and a vehicle identification code area in that vehicle exterior area, and outputs the data as vehicle exterior area identification data and vehicle identification code area identification data, respectively. The vehicle identification code area tracking means 75 outputs, based on the vehicle exterior area estimated by the first AI estimation means 971 and its image data, d1) vehicle exterior area tracking result data generated by performing object tracking processing on images in the vehicle exterior area between previous and next frames of the time-series still image data, and d2) vehicle identification code area tracking result data generated by object tracking on images in the vehicle identification code area by performing object tracking processing on images in the vehicle identification code area. If the vehicle identification code area exists within the vehicle identification code area in a specific frame, the temporary identifier assignment means 977 assigns a vehicle temporary identifier common to the vehicle identification code area and the vehicle identification code area. At this time, if at least one of the vehicle exterior area tracking result data and the vehicle specific code area tracking result data has been used to track the vehicle exterior area or the vehicle specific code area of ​​the specific frame in a frame other than the specific frame, the temporary vehicle identifier to be assigned is the temporary vehicle identifier assigned to the vehicle exterior area or the vehicle specific code area of ​​the specific frame. Other means are the same as those in the first embodiment.

[0237] In this embodiment, the correct answer data are provided as linked and both-containing data, unlinked and single data (only vehicle body region specifying data or only identifier region specifying data), and both-non-containing data. These three patterns of images and their correct answer data are provided to the first estimation program 26f as training data, and the first estimation program 26f learns. Note that this correct answer data is created by a human referring to the image data and determining which areas are the vehicle body frame 141 and the license plate frame 142.

[0238] Figure 22 shows a functional block diagram of the AI ​​estimation device 970 of Figure 21 during learning. The difference from Figure 19 is the first supervised data provided to the first AI estimation means 971. Specifically, the supervised data provided are linked, both-containing data, unlinked single data (only vehicle body region specifying data or only identifier region specifying data), and both-not-containing data, and learning is performed by providing these three patterns of images and their supervised data as training data.

[0239] The first correct answer data is created by a person who refers to the image data and determines which areas are the vehicle body frame 141 and the license plate frame 142.

[0240] The estimation result data output from the first AI estimation means 971 is one of the three types described above, and the process of extracting an image from such estimation result data and estimating a vehicle identification identifier by the second AI estimation means is the same as in the first embodiment.

[0241] (10. Other Embodiments) In the above embodiment, the image extraction means 403 does not provide the vehicle identification code area image data to the second AI estimating means 401 if the first AI estimating device 200 does not extract a vehicle identification code area. However, the vehicle identification code area image data may be provided to the second AI estimating means 401 to learn such cases. In this case, learning may be performed so that different outputs are output, such as "null" for "no vehicle identification code" and "unclear" for "a vehicle identification code is present but unclear."

[0242] In the above embodiment, the license plate frame is extracted separately from the vehicle frame, so that for example, tires, headlights on corners, white boxes on trucks, signs, etc. are not mistakenly recognized as the license plate frame.

[0243] In addition, in the case of a car transporter that carries one or more cars, it is necessary to identify the license plate taking into account the relationship between the vehicle frame and the license plate frame in the image. This technology can also learn such correspondences.

[0244] In addition, in this embodiment, both the license plate frame and the vehicle body frame are tracked, and the vehicle being photographed is identified by a temporary vehicle ID that links the two.

[0245] Here, tracking with the license plate frame is quite difficult because the image area is small.

[0246] By managing the vehicle with a temporary vehicle ID in this way, even if the vehicle is lost and a new license plate ID is assigned, it is not necessary to assign a new temporary vehicle ID each time. This means that, for example, even if a new vehicle is detected, it is not necessary to send the information to the server each time. This reduces the amount of data transmitted.

[0247] Note that color images and black-and-white images (IR images, also known as night vision) can be used as training data. In this case, black-and-white images can be intentionally generated from color images and used as training data.

[0248] It is also possible to set a minimum size limit for the size of the vehicle or license plate, and ignore any smaller size. Also, regarding how much of the license plate of a vehicle that is outside the image should be input, for example, it is possible to input only up to the point where the characteristics of the vehicle or license plate can be confirmed from the image.

[0249] In this embodiment, whether the vehicle number is clear or not is determined for each character. Therefore, there are various patterns of detected vehicle numbers, such as "Shiga???1234", "????A1234", and "Shiga330?1234".

[0250] During training of the first estimation program 126f, even if the characters in the license plate frame are unclear and ultimately unrecognizable, the license plate frame is provided as training data as correct answer data, so that the first estimation program can extract the license plate frame even if the characters in the license plate frame are unclear.

[0251] In the above embodiment, a temporary vehicle ID is assigned that links the license plate ID and the vehicle ID, and this is linked to the number estimated by the second estimation program 26s. Therefore, even if either one is lost, as long as the connection is made via either the license plate ID or the vehicle ID, the same temporary vehicle ID can be used for management. Furthermore, the license plate frame may be lost midway between multiple frames. For example, this may occur if the number portion is unclear or cut off. In such a case, if the license plate is lost and then restored, a new number ID is assigned. However, even in this case, the number is ultimately recognized as the number of the temporary vehicle ID. The same applies when the vehicle is lost.

Claims

1. A) A vehicle identification code determination device that, when given frame image data in which still images including a vehicle's outer shape area and / or a vehicle identification code area are arranged in time series, estimates a character string in a vehicle identification code area of ​​a certain vehicle from the frame image data, B) a first AI estimation means for estimating a vehicle outer shape area and a vehicle specific code area in each frame image data when the frame image data is given as input data, and outputting the vehicle outer shape area and vehicle specific code area as vehicle outer shape area identification data and vehicle specific code area identification data, respectively; C) a second AI estimation means for extracting an image of the vehicle identification code area from the vehicle identification code area identification data estimated by the first AI estimation means and image data including the vehicle outer shape, and estimating the vehicle identification code; D) A tracking means for outputting the following vehicle outer shape area tracking result data and vehicle identification code area tracking result data as area tracking data based on the vehicle outer shape area estimated by the first AI estimation means and its image data; d1) Vehicle outer shape region tracking result data generated by performing object tracking processing on images in the vehicle outer shape region between previous and next frames of the time-series still image data; d2) vehicle identification code area tracking result data generated by object tracking of the image in the vehicle identification code area, which performs object tracking processing of the image in the vehicle identification code area between previous and next frames of the time-series still image data; E) an area linking means for linking the vehicle outer shape area and the vehicle specific code area in each frame when the vehicle outer shape area and the vehicle specific code area overlap each other, and outputting the result of linking as intra-frame linking result data; F) A means for assigning a temporary vehicle identifier that identifies the vehicle in the linked vehicle outer shape area and vehicle specific code area in each frame, wherein when determining the temporary vehicle identifier to be assigned, if at least one of the vehicle outer shape area or the vehicle specific code area assigned in a specific frame can be tracked using the area tracking data, a temporary identifier assignment means assigns the same temporary vehicle identifier as the temporary vehicle identifier in the specific frame; In a vehicle identification code determination device equipped with G) The case where the vehicle specific code area is in an overlapping relationship means that the vehicle specific code area is located below the vehicle outer shape area, and the overlap between the vehicle outer shape area and the vehicle specific code area is 80% or more. A vehicle identification code determination device characterized by the above.

2. A) A vehicle identification code determination device that, when given frame image data in which still images including a vehicle's outer shape area and / or a vehicle identification code area are arranged in time series, estimates a character string in a vehicle identification code area of ​​a certain vehicle from the frame image data, B) a first AI estimation means for estimating a vehicle exterior area in each frame image data and a vehicle specific code area in the vehicle exterior area when the frame image data is given as input data, and outputting the vehicle exterior area identification data and the vehicle specific code area identification data, respectively; C) a second AI estimation means for extracting an image of the vehicle identification code area from the vehicle identification code area identification data estimated by the first AI estimation means and image data including the vehicle outer shape, and estimating the vehicle identification code; D) A tracking means for outputting the following vehicle outer shape area tracking result data and vehicle specific code area tracking result data based on the vehicle outer shape area estimated by the first AI estimation means and its image data: d1) Vehicle outer shape region tracking result data generated by performing object tracking processing on images in the vehicle outer shape region between previous and next frames of the time-series still image data; d2) vehicle identification code area tracking result data generated by object tracking of the image in the vehicle identification code area, which performs object tracking processing of the image in the vehicle identification code area between previous and next frames of the time-series still image data; E) an area linking means for linking the vehicle outer shape area and the vehicle specific code area in each frame when the vehicle outer shape area and the vehicle specific code area overlap each other, and outputting the result of linking as intra-frame linking result data; F) A means for assigning a temporary vehicle identifier that identifies the vehicle in the linked vehicle outer shape area and vehicle specific code area in each frame, wherein when determining the temporary vehicle identifier to be assigned, if at least one of the vehicle outer shape area or the vehicle specific code area assigned in a specific frame can be tracked using the area tracking data, a temporary identifier assignment means assigns the same temporary vehicle identifier as the temporary vehicle identifier in the specific frame; In a vehicle identification code determination device equipped with G) The case where the vehicle specific code area is in an overlapping relationship means that the vehicle specific code area is located below the vehicle outer shape area, and the overlap between the vehicle outer shape area and the vehicle specific code area is 80% or more. A vehicle identification code determination device characterized by the above. A vehicle identification code determination device comprising:

3. The vehicle identification code determination device of claim 1 or 2, further comprising: a determination means for determining the vehicle identification code estimated by the second AI estimation means as the vehicle identification code of the vehicle identified by the temporary vehicle identifier; A vehicle identification code determination device characterized by the above.

4. In the vehicle identification code determination device of claim 3, The second AI estimation means estimates a code indicating the character string when the character string is unclear at the time of estimating the vehicle identification code, the determination means, when a part or all of the vehicle identification code estimated by the second AI estimation means for a vehicle identified by the temporary vehicle identifier is a code indicating blurring and therefore differs depending on the frame, determines the vehicle identification code of a frame with fewer codes indicating blurring as the vehicle identification code of the vehicle identified by the temporary vehicle identifier; A vehicle identification code determination device characterized by the above.

5. In the vehicle identification code determination device of claim 3, the determining means determines whether the moving direction of the vehicle is an approaching direction or a non-approaching direction based on the positional relationship of each vehicle specific code area between the front and rear frames determined by the tracking means, or the positional relationship of each vehicle specific code area, for the vehicle specific code in the time-series still image data estimated by the second estimating means, and adopts the vehicle specific code area determined in the rear frame if the moving direction of the vehicle is an approaching direction, and adopts the vehicle specific code area determined in the front frame if the moving direction is a non-approaching direction; A vehicle identification code determination device characterized by the above.

6. In the vehicle identification code determination device of claim 3, the determination means, when a character string represented by the vehicle identification code estimated by the second AI estimation means is unclear in one of the front and rear frames, and therefore vehicle identification codes assigned with the same temporary vehicle identifier do not match in the plurality of frames, identifies a character string in the other frame different from the one of the front and rear frames as the vehicle identification code; A vehicle identification code determination device characterized by the above.

7. In the vehicle identification code determination device of claim 3, The vehicle identification code is composed of a group of characters: "region name," "classification number," "hiragana," and "series designation number," determining that each of the character groups is represented by a vehicle identification code estimated by the second AI estimation means; A vehicle identification code determination device characterized by the above.

8. In the vehicle identification code determination device of claim 6, the determining means determines the vehicle identification code by comparing the character string in the one or the other frame for each character of the character group; A vehicle identification code determination device characterized by the above.

9. A vehicle identification code determination method in which, when frame image data in which still images including a vehicle's outer shape area and / or a vehicle identification code area are arranged in time series is provided, a computer estimates a character string in the vehicle identification code area of ​​a certain vehicle from the frame image data, the method comprising the steps of: a first AI estimation step of estimating a vehicle outer shape area and a vehicle specific code area in each frame image data when the frame image data is given as input data, and outputting the vehicle outer shape area and the vehicle specific code area as vehicle outer shape area identification data and vehicle specific code area identification data, respectively; a second AI estimation step of extracting an image of the vehicle specific code area from image data including the vehicle specific code area identification data estimated in the first AI estimation step and the vehicle outer shape, and estimating the vehicle specific code; a tracking step of outputting the following 1) vehicle outer region tracking result data and 2) vehicle identification code region tracking result data as region tracking data based on the vehicle outer region estimated by the first AI estimation step and its image data; 1) Vehicle outer shape region tracking result data generated by performing object tracking processing on images in the vehicle outer shape region between previous and next frames of the time-series still image data; 2) Vehicle identification code area tracking result data generated by object tracking of the image in the vehicle identification code area, which performs object tracking processing of the image in the vehicle identification code area between previous and next frames of the time-series still image data; E) an area linking step of linking the vehicle exterior area and the vehicle specific code area in each frame when the vehicle exterior area and the vehicle specific code area overlap each other and outputting the result of linking as intra-frame linking result data; F) a step of assigning a temporary vehicle identifier that identifies the vehicle in the linked vehicle outer shape area and vehicle specific code area in each of the frames, wherein, when determining the temporary vehicle identifier to be assigned, if at least one of the vehicle outer shape area or the vehicle specific code area assigned in a specific frame can be tracked using the area tracking data, a temporary identifier assignment step of assigning the same temporary vehicle identifier as the temporary vehicle identifier in the specific frame; A vehicle identification code determination method comprising: G) The case where the vehicle specific code area is in an overlapping relationship means that the vehicle specific code area is located below the vehicle outer shape area, and the overlap between the vehicle outer shape area and the vehicle specific code area is 80% or more. A vehicle identification code determination method characterized by the above.

10. A vehicle identification code determination method in which, when frame image data in which still images including an outer shape region and / or a vehicle identification code region of a vehicle are arranged in time series is provided, a computer estimates a character string in a vehicle identification code region of a certain vehicle from the frame image data, a first AI estimation step of, when the frame image data is given as input data, estimating a vehicle exterior area in each frame image data and a vehicle specific code area in that vehicle exterior area, and outputting the vehicle exterior area identification data and the vehicle specific code area identification data, respectively; a second AI estimation step of extracting an image of the vehicle specific code area from image data including the vehicle specific code area identification data estimated in the first AI estimation step and the vehicle outer shape, and estimating the vehicle specific code; a tracking step of outputting the following 1) vehicle outer region tracking result data and 2) vehicle identification code region tracking result data as region tracking data based on the vehicle outer region estimated by the first AI estimation step and its image data; 1) Vehicle outer shape region tracking result data generated by performing object tracking processing on images in the vehicle outer shape region between previous and next frames of the time-series still image data; 2) Vehicle identification code area tracking result data generated by object tracking of the image in the vehicle identification code area, which performs object tracking processing of the image in the vehicle identification code area between previous and next frames of the time-series still image data; E) an area linking step of linking the vehicle exterior area and the vehicle specific code area in each frame when the vehicle exterior area and the vehicle specific code area overlap each other and outputting the result of linking as intra-frame linking result data; F) A temporary identifier assigning step of assigning a temporary vehicle identifier that identifies the vehicle in the linked vehicle outer shape area and vehicle specific code area in each frame, wherein, when determining the temporary vehicle identifier to be assigned, if at least one of the vehicle outer shape area or the vehicle specific code area assigned in a specific frame can be tracked using the area tracking data, the temporary vehicle identifier is assigned to the vehicle in the specific frame; Equipped with G) The case where the vehicle specific code area is in an overlapping relationship means that the vehicle specific code area is located below the vehicle outer shape area, and the overlap between the vehicle outer shape area and the vehicle specific code area is 80% or more. A vehicle identification code determination method characterized by the above.

Images