Information identification method and device, and storage medium

Abstract

The application provides an information recognition method and device and a storage medium, the method comprising: collecting a video stream of a predetermined area in real time, and extracting N starting image frames corresponding to a current time from the video stream based on a sliding window method; if a first object in the N starting image frames is detected to be in a starting state, performing a de-duplication optimization process on characters in a to-be-tested area image in each image frame to obtain a character recognition result, until the first object in the N ending image frames is detected to be in an ending state, and a plurality of character recognition results are obtained; wherein the N ending image frames are extracted at an ending time after any time length from the current time in the video stream; the image frames belong to a plurality of image frames extracted in the video stream from the current time to the ending time; and a target character recognition result is determined based on the plurality of character recognition results. The recognition accuracy of license plate information is improved.

Description

Technical Field

[0001] The present invention relates to the field of information recognition technology, and in particular to an information recognition method, device and storage medium. Background Technology

[0002] In smart logistics parks, efficient collaboration between vehicle management and loading / unloading management is crucial to ensuring efficient logistics operations. License plate recognition is a vital component of vehicle management, and it operates across various scenarios, such as license plate recognition at gates and platforms, typically employing computer vision methods. Academic solutions generally aim for accurate identification of license plates appearing in a single image, but they fail to consider the need for continuous monitoring of surveillance footage in real-world scenarios, resulting in poor handling of interference. Existing industrial solutions primarily focus on recognizing the front license plates of vehicles, with cameras positioned close to the plates, and the scenarios are relatively simple and straightforward. Therefore, existing solutions exhibit lower accuracy in recognizing license plate information in more complex scenarios with greater interference. Summary of the Invention

[0003] The present invention provides an information recognition method, device and storage medium that can improve the accuracy of vehicle license plate information recognition.

[0004] The technical solution of this invention is implemented as follows:

[0005] This invention provides an information identification method, including:

[0006] The system acquires video streams from a predetermined area in real time and extracts N initial image frames corresponding to the current moment from the video stream using a sliding window method; N is an integer greater than 1.

[0007] If the first object is detected as the initial state in the N initial image frames, then the characters in the test area image of each image frame are deduplicated and optimized to obtain character recognition results, until the first object is detected as the final state in the N final image frames, resulting in multiple character recognition results; wherein, the N final image frames are extracted from the video stream at a termination time after any time interval from the current time; each image frame belongs to multiple image frames extracted from the video stream from the current time to the termination time;

[0008] Based on the recognition results of the multiple characters, the recognition result of the target character is determined.

[0009] In the above scheme, if the first object is detected as the starting state in the N initial image frames, then the characters in the test area image of each image frame are deduplicated and optimized to obtain character recognition results, until the first object is detected as the ending state in the N termination image frames, thus obtaining multiple character recognition results, including:

[0010] The initial detection results are obtained by detecting the N initial image frames;

[0011] If the initial detection result indicates that the first object is close to or far from the target area, then each of the image frames extracted from the video stream is processed starting from the first image frame among the N initial image frames to obtain the corresponding image of the area to be tested.

[0012] The image of the region to be tested is processed using a preset target detection model to obtain multiple characters and their corresponding character-related information;

[0013] The multiple characters are deduplicated and optimized using the character-related information to obtain the character recognition result composed of characters in a certain order. The process continues until the termination detection result indicates that the first object is in a state of no displacement change or that there is no first object at the termination time. The multiple character recognition results are obtained by detecting the N termination image frames.

[0014] In the above scheme, the step of detecting the N initial image frames to obtain the initial detection result includes:

[0015] The N initial image frames are processed using a first preset detection model to obtain the initial detection results;

[0016] If the initial detection result indicates that the first object is close to or far from the target area, then each of the N initial image frames is processed starting from the first image frame extracted from the video stream to obtain the corresponding image of the area to be tested, including:

[0017] If the initial detection result indicates that at least M of the N initial image frames include the first object, and the distance between the first object and the target region is increasing or decreasing, then each image frame is processed by the second preset detection model to obtain the image of the region to be tested; M is an integer greater than 1 and less than N.

[0018] In the above scheme, the step of processing the image of the region to be tested using a preset target detection model to obtain multiple characters and their corresponding character-related information includes:

[0019] The image of the region to be tested is processed using a preset target detection model to obtain the multiple characters, as well as the character category probability information and character box information corresponding to the multiple characters.

[0020] In the above scheme, the deduplication optimization processing of the multiple characters using the character-related information to obtain the character recognition result composed of characters in a certain order, until the termination detection result indicates that the first object is in a state of no displacement change or the first object is not present at the termination time, thus obtaining the multiple character recognition results, includes:

[0021] An overlap matrix is ​​calculated using information from multiple character boxes, and the multiple characters are deduplicated using the overlap matrix to obtain multiple first characters.

[0022] By combining the first character box information corresponding to the multiple first characters, abnormal characters are removed from the multiple first characters to obtain multiple second characters;

[0023] Using the second character box information corresponding to the multiple second characters respectively, the multiple second characters are sorted in layers to obtain the character recognition result. The process continues until the termination detection result indicates that the distance between the first object and the target region remains unchanged in at least M of the N termination image frames, or the first object is included in K of the N starting image frames, thus obtaining the multiple character recognition results; K is an integer greater than or equal to 1 and less than M.

[0024] In the above scheme, the overlap matrix is ​​calculated using information from multiple character boxes, and the overlap matrix is ​​used to perform deduplication on the multiple characters to obtain multiple first characters, including:

[0025] Using the area information of multiple character frames included in the multiple character frame information, the overlap between any two characters is calculated, and the overlap matrix is ​​constructed using the obtained multiple overlaps.

[0026] The multiple overlap degrees are compared sequentially with preset thresholds to obtain the comparison result for each overlap degree;

[0027] If the comparison result indicates that the first overlap is greater than the preset threshold, then the rows and columns corresponding to the overlapping characters in the overlap matrix are deleted to obtain a new overlap matrix, until the overlap in the overlap matrix is ​​less than the preset threshold to obtain the target matrix; the overlapping character is the character with the lowest probability of the character category corresponding to the first overlap.

[0028] The plurality of first characters are determined by the characters corresponding to the degree of overlap in any row of the target matrix.

[0029] In the above scheme, the multiple characters include: T characters; the multiple character box area information includes: T character box area information; T is an integer greater than 1;

[0030] The step of calculating the overlap between any two characters using the area information of the multiple character frames, and constructing the overlap matrix using the obtained overlap values, includes:

[0031] Based on the character frame area information of the first character to the character frame area information of the Tth character, calculate the first group of T overlaps between the first character and the first to the Tth characters respectively;

[0032] The first row of the overlap matrix is ​​constructed using the first group of T overlaps, until the Tth character is calculated to have the Tth group of T overlaps with the first character to the Tth character respectively. The last row of the overlap matrix is ​​constructed using the Tth group of T overlaps, thus obtaining the overlap matrix.

[0033] In the above scheme, by combining the first character box information corresponding to the multiple first characters, abnormal characters are removed from the multiple first characters to obtain multiple second characters, including:

[0034] Multiple first character box information are processed by a preset anomaly detection model to remove the abnormal character with the lowest density among the multiple first characters, thereby obtaining the multiple second characters.

[0035] In the above scheme, the step of using the second character box information corresponding to the plurality of second characters to perform hierarchical sorting processing on the plurality of second characters to obtain the character recognition result includes:

[0036] A fitted line is obtained by linear fitting of multiple center point information in multiple second character box information, and the sum of squared errors of the multiple center point information from the fitted line is calculated.

[0037] The number of character layers in the character recognition result is determined based on the magnitude of the sum of squared errors;

[0038] If the character layer has two layers, then extract multiple character combinations including any P second characters from the multiple second characters; P is an integer greater than 1 and less than T;

[0039] Linear fitting is performed on each of the multiple character combinations, and multiple sums of squared errors are calculated. The P third characters included in the character combination corresponding to the smallest sum of squared errors are then determined.

[0040] The remaining characters are concatenated according to the order of their corresponding horizontal coordinates in the second character box information, and then the P third characters are concatenated according to the order of their corresponding horizontal coordinates to obtain the character recognition result; the remaining characters are the characters other than the P third characters among the plurality of second characters.

[0041] In the above scheme, after determining the number of character layers in the character recognition result based on the magnitude of the sum of squared errors, the method further includes:

[0042] If the character layer is one layer, the multiple second characters are concatenated according to the size order of their respective horizontal coordinate information to obtain the character recognition result.

[0043] In the above scheme, determining the target character recognition result based on the multiple character recognition results includes:

[0044] The character category probability information corresponding to each character in the multiple character recognition results is added together to obtain multiple total probabilities;

[0045] The character recognition result corresponding to the largest total probability among the multiple total probabilities is determined as the target character recognition result.

[0046] In the above scheme, the sliding window-based method extracts the N starting image frames corresponding to the current moment from the video stream, including:

[0047] Extract the current image frame corresponding to the current moment from the video stream;

[0048] Starting from the current image frame in the video stream, an image frame is extracted along the time axis at predetermined intervals or predetermined number of image frames until N-1 image frames are extracted.

[0049] The current image frame and the N-1 image frames are combined along the time axis to obtain the N initial image frames.

[0050] This invention also provides an information identification device, comprising:

[0051] The acquisition and extraction unit is used to acquire video streams of a predetermined area in real time, and extract N starting image frames corresponding to the current moment from the video stream based on a sliding window method; N is an integer greater than 1.

[0052] The detection processing unit is configured to, if a first object is detected as a starting state in any of the N initial image frames, perform deduplication optimization processing on the characters in the test region image of each image frame to obtain a character recognition result, until the first object is detected as a ending state in any of the N ending image frames, thereby obtaining multiple character recognition results; wherein, the N ending image frames are extracted from the video stream at an ending time after any time interval from the current time; each image frame belongs to multiple image frames extracted from the video stream from the current time to the ending time;

[0053] The determining unit is used to determine the target character recognition result based on the multiple character recognition results.

[0054] This invention also provides an information identification device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in the above method.

[0055] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method.

[0056] In this embodiment of the invention, a video stream of a predetermined area is acquired in real time, and N initial image frames corresponding to the current moment are extracted from the video stream based on a sliding window method; N is an integer greater than 1; if the first object is detected as the initial state in the N initial image frames, the characters in the image of the region to be tested in each image frame are deduplicated and optimized to obtain character recognition results, until the first object is detected as the final state in the N final image frames, resulting in multiple character recognition results; wherein, the N final image frames are extracted from the video stream at a final moment after any time interval from the current moment; the image frames belong to multiple image frames extracted from the video stream from the current moment to the final moment; based on the multiple character recognition results, the target character recognition result is determined. Since this scheme recognizes license plate information by capturing the period between the arrival and departure of the first object, and simultaneously performs deduplication and optimization processing on the recognized characters, it overcomes the difficulty of diverse formats in the license plate recognition process, thereby improving the recognition accuracy of license plate information. Attached Figure Description

[0057] Figure 1 A schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention;

[0058] Figure 2 This is an optional effect diagram of the information recognition method provided in the embodiment of the present invention;

[0059] Figure 3A schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention;

[0060] Figure 4 This is an optional effect diagram of the information recognition method provided in the embodiment of the present invention;

[0061] Figure 5 This is an optional effect diagram of the information recognition method provided in the embodiment of the present invention;

[0062] Figure 6 This is an optional effect diagram of the information recognition method provided in the embodiment of the present invention;

[0063] Figure 7 This is an optional effect diagram of the information recognition method provided in the embodiment of the present invention;

[0064] Figure 8 A schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention;

[0065] Figure 9 A schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention;

[0066] Figure 10 This is an optional effect diagram of the information recognition method provided in the embodiment of the present invention;

[0067] Figure 11 A schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention;

[0068] Figure 12 A schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention;

[0069] Figure 13 A schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention;

[0070] Figure 14 This is an optional effect diagram of the information recognition method provided in the embodiment of the present invention;

[0071] Figure 15 This is a schematic diagram of the structure of the information recognition device provided in an embodiment of the present invention;

[0072] Figure 16 This is a schematic diagram of a hardware entity of the information recognition device provided in an embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0075] If similar descriptions such as "first / second" appear in the invention document, the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the invention described herein can be implemented in an order other than that illustrated or described herein.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0077] Vehicle management is a crucial aspect of logistics. Effective management and scheduling of vehicles ensures the safe and efficient flow of goods and guarantees the quality of logistics services. Accurate vehicle information identification is a fundamental prerequisite for resource allocation in logistics scenarios. For example, vehicles entering a logistics park for loading and unloading must strictly adhere to a pre-established schedule. Identifying vehicle information helps confirm whether vehicles arrive on time and can begin operations as planned. Furthermore, vehicle and cargo information must be strictly matched. Warehouses need to begin loading / unloading preparations upon vehicle arrival and allocate loading docks based on resource availability. License plate recognition at loading docks helps verify vehicle information and confirm that vehicles have arrived at the correct loading / unloading locations. In traditional production processes, workers manually record each vehicle's license plate number. In today's intelligent and information-driven era, this traditional recording method results in significant waste of human and financial resources, while slowing down operational efficiency and throughput. Therefore, automatic license plate recognition systems are widely used. They not only quickly and accurately identify vehicle license plate numbers, saving manpower, but also support the implementation of automated scheduling systems, and the digitized data is easier to store.

[0078] Currently, automatic license plate recognition is generally achieved using computer vision techniques. Most vision-based automatic license plate recognition algorithms typically consist of two steps: first, license plate detection, which involves locating the pixel position of the license plate in an image; and second, character recognition, which identifies the content of each character in the license plate to assemble the license plate. Current computer vision-based automatic license plate recognition systems can be categorized into academic and industrial solutions. Academic solutions generally aim for accurate recognition of license plates appearing in a single image, but they do not consider the need for continuous recognition of surveillance footage in real-world scenarios and have poor resistance to interference. Existing industrial solutions are mostly used to recognize the front license plates of vehicles where the camera is close to the license plate, and the scenarios are relatively simple, such as license plate recognition systems in parking lots. In loading and unloading areas like platforms, vehicles often reverse into the space for ease of loading and unloading. Therefore, cameras primarily capture the rear license plates. Compared to the uniform single-line format of the front license plates, rear license plates can be either single-line or double-line (yellow plates are double-line). Furthermore, rear license plates in logistics scenarios are often dirtier than front license plates, with the characters frequently obscured by dust. Additionally, in real logistics environments, cameras are often fixed at higher positions, resulting in smaller license plates that are easily obscured and affected by lighting and angle. Consequently, the accuracy of conventional automatic license plate recognition systems is very low in this scenario.

[0079] This invention provides an information identification method; please refer to [link / reference]. Figure 1 This is an optional flowchart illustrating an information recognition method provided in an embodiment of the present invention, which will be combined with... Figure 1 The steps shown are explained.

[0080] S101. Real-time acquisition of video stream from a predetermined area, and extraction of N initial image frames corresponding to the current moment from the video stream using a sliding window method.

[0081] In this embodiment of the invention, the information recognition device acquires video streams of a predetermined area in real time, and extracts N initial image frames corresponding to the current moment from the video stream based on a sliding window method. Here, N is an integer greater than 1.

[0082] In this embodiment of the invention, the information recognition device acquires video streams of a predetermined area using a camera positioned within that area. The information recognition device extracts one image frame every predetermined number of image frames and uses a sliding window method to determine the N starting image frames corresponding to the current moment.

[0083] In this embodiment of the invention, the information recognition device extracts the image frame corresponding to the current moment from the video stream, and extracts multiple image frames at intervals along the time axis to obtain N initial image frames.

[0084] In this embodiment of the invention, the information recognition device can be a terminal or a server connected to a camera in a predetermined area.

[0085] S102. If the first object is detected in N initial image frames as the initial state, then the characters in the image of the region to be tested in each image frame are deduplicated and optimized to obtain the character recognition result. This process continues until the first object is detected in N final image frames as the final state, resulting in multiple character recognition results.

[0086] In this embodiment of the invention, if the information recognition device detects that the first object in one of the N initial image frames is the initial state, it performs deduplication optimization processing on the characters in the image of the region to be tested in each image frame to obtain the character recognition result, until it stops when it detects that the first object in one of the N final image frames is the final state, thus obtaining multiple character recognition results.

[0087] Among them, the N terminating image frames are extracted from the video stream at the terminating time after any time interval from the current time; the image frames are multiple image frames extracted from the video stream from the current time to the terminating time.

[0088] In this embodiment of the invention, if the information recognition device detects that the first object in any of the N initial image frames is either close to or far from the target area, the information recognition process is triggered. The information recognition device processes each image frame extracted from the video stream from the current time to the end time, extracting the characters within the test area image of each image frame and their corresponding character-related information. The information recognition device can perform deduplication optimization processing on multiple characters using the character-related information to obtain the character recognition result corresponding to each image frame. After detecting image frames for a certain period, the information recognition device stops when it detects that the first object in any of the N end image frames at any end time is stationary or that there is no first object in the N end image frames, thus obtaining multiple character recognition results.

[0089] For example, if the information recognition device detects a change in the Y-axis coordinate of the first object in N initial image frames, it processes each image frame extracted from the video stream starting from the current moment, extracting the characters in the test area image of each image frame and their corresponding character-related information. The information recognition device can perform deduplication optimization processing on multiple characters using the character-related information to obtain the character recognition result corresponding to each image frame, until the information recognition device detects that the first object in the N termination image frames corresponding to the termination time one minute after the current moment is stationary, thus obtaining multiple character recognition results.

[0090] In this embodiment of the invention, the license plate information recognition process mainly includes: 1. Vehicle body detection. 2. Vehicle status judgment. 3. License plate detection. 4. License plate character recognition. Vehicle body detection is used to confirm whether there is a vehicle in the platform area, and then to determine whether a vehicle has arrived at or left the platform by detecting changes in the vehicle's position. Since the license plate is also one of the significant features of a vehicle, this solution can also be simplified to using license plate detection to determine whether there is a vehicle in the platform area, and then to determine whether a vehicle has arrived at or left the platform by changing the position of the license plate. However, compared to the vehicle body, the license plate is a smaller target, more difficult to detect, and may be obscured. Therefore, this solution first uses vehicle body detection to determine whether there is a vehicle in the platform area. Finally, this solution also proposes an engineering solution based on edge deployment. Specifically:

[0091] In this embodiment of the invention, the information recognition device uses target detection methods from computer vision to solve the vehicle detection problem, determining whether there are vehicles in the platform area through video streams captured by a camera. To effectively identify vehicles entering and leaving the platform area and distinguish them from vehicles from other angles, the information recognition device can identify the rear cargo compartment of the vehicle. Here, a deep learning-based method is used for detection. During the training process of the deep learning model, the rear cargo compartment of the vehicle needs to be labeled with rectangular boxes, such as... Figure 2 As shown. The trunk of the vehicle can be labeled as the target object, and the labeled data is used for training. Since object detection is a relatively mature technology in the field of computer vision, there are various techniques to choose from (You Only Look Once, YOLO series algorithms, Region-CNN, R-CNN algorithm, CenterNet algorithm). In this embodiment of the invention, YOLOv5 is selected; in other embodiments, other algorithms can also be used.

[0092] The information recognition device determines the status of vehicles and platform occupancy in the current platform area by detecting changes in vehicle positions. To ensure accurate and detailed assessment of vehicle and platform conditions, in this embodiment of the invention, the information recognition device can summarize the identified information into vehicle status, platform status, and events. Vehicle status is categorized as no vehicle, vehicle entering the platform, vehicle leaving the platform, and vehicle parked. Platform status is categorized as occupied and released. Events include vehicle arrival at the platform, vehicle departure from the platform, vehicle position adjustment, and temporary vehicle parking. The determination method is as follows:

[0093] The information recognition device can determine the position of the rear carriage based on changes detected by a first preset detection model. To achieve this, a monitoring camera needs to periodically capture images of a predetermined area (platform area) (using a 5-frame-per-second method) and detect the vehicle's position in each frame. The information recognition device uses the vehicle's position in the current image frame and the N-1 closest image frames to it for each judgment. If the vehicle is detected in at least M of the N initial image frames, and its position is close to the platform, the vehicle is determined to be in the parking lot. If the vehicle is far from the platform, it is determined to be in the exiting phase. If the vehicle is detected in at least M image frames, but its position does not change significantly, it is in a parked state. If the vehicle is detected in fewer than M frames, it is in a no-vehicle state. Since vehicle recognition may involve misidentification and missed identification, anomaly detection must be performed before dynamic judgment to eliminate interference from misidentification and missed identification.

[0094] The information recognition device's judgment of platform status and event recognition need to be complementary. In the initial state of system startup, the platform status can be configured according to the actual situation: if there is a vehicle on the platform, it is in an occupied state; if there is no vehicle, it is in a released state. Event reporting requires combining the vehicle and platform status. An event is defined by a start state and a stop state. Approaching the platform and moving away from the platform are start states; no vehicle and parked are stop states. Once the information recognition device detects a start state, it triggers event monitoring to begin license plate information recognition for each image frame. If a stop state is detected after a start state has been detected, a complete event is identified. Combining the platform and vehicle status, the event definition and corresponding platform status update are as follows:

[0095] 1. When the platform status is released, the vehicle status changes from the parking status to the stationary status when the vehicle arrives at the platform. After the event ends, the platform status needs to be updated to occupied.

[0096] 2. When the platform status is released, the change of vehicle status from inbound to no vehicle status is a temporary vehicle parking event. After the event ends, the platform status will still be released.

[0097] 3. When the platform is occupied, the vehicle status changes from inbound / outbound to parked status, which is a vehicle adjustment event. After the event ends, the platform status remains occupied.

[0098] 4. When the platform status is occupied, the vehicle status changes from inbound / outbound to no vehicle status as a vehicle leaving the platform event. After the event ends, the platform status is updated to released.

[0099] 5. Combinations of other states are invalid events and should be discarded directly without reporting.

[0100] In this embodiment of the invention, the information recognition device can also generate corresponding identification information for each event. After the information recognition device recognizes the target character recognition result, it can map and store the target character recognition result with the corresponding event identification information.

[0101] S103. Based on the recognition results of multiple characters, determine the recognition result of the target character.

[0102] In this embodiment of the invention, the information recognition device determines the target character recognition result based on multiple character recognition results.

[0103] In this embodiment of the invention, when the information recognition device identifies each character recognition result, it generates character category probability information for all characters in that character recognition result. The information recognition device can determine the character recognition result with the largest sum of character category probability information for all characters as the target character recognition result.

[0104] In this embodiment of the invention, a video stream of a predetermined area is acquired in real time, and N initial image frames corresponding to the current moment are extracted from the video stream based on a sliding window method; N is an integer greater than 1; if the first object is detected as the initial state in the N initial image frames, the characters in the image of the region to be tested in each image frame are deduplicated and optimized to obtain character recognition results, until the first object is detected as the final state in the N final image frames, resulting in multiple character recognition results; wherein, the N final image frames are extracted from the video stream at a final moment after any time interval from the current moment; the image frames belong to multiple image frames extracted from the video stream from the current moment to the final moment; based on the multiple character recognition results, the target character recognition result is determined. Since this scheme recognizes license plate information by capturing the period between the arrival and departure of the first object, and simultaneously performs deduplication and optimization processing on the recognized characters, it overcomes the difficulty of diverse formats in the license plate recognition process, thereby improving the recognition accuracy of license plate information.

[0105] In some embodiments, see Figure 3 , Figure 3 This is a schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention. Figure 1 The shown S102 can also be implemented by S104 to S107, which will be explained in conjunction with each step.

[0106] S104. Detect the N initial image frames to obtain the initial detection results.

[0107] In this embodiment of the invention, the information recognition device detects N initial image frames to obtain initial detection results.

[0108] In this embodiment of the invention, the information recognition device can use the YOLOv5 model to detect N initial image frames to obtain the initial detection result. In other embodiments, other detection models may also be used, and this embodiment of the invention does not impose any limitations.

[0109] S105. If the initial detection result indicates that the first object is close to or far from the target area, then each image frame extracted from the video stream is processed starting from the first image frame among the N initial image frames to obtain the corresponding image of the area to be tested.

[0110] In this embodiment of the invention, if the information recognition device detects that the initial detection result indicates that the first object is close to or far from the target area, then it processes each image frame extracted from the video stream starting from the first image frame among the N initial image frames to obtain the corresponding image of the area to be tested.

[0111] In this embodiment of the invention, if the information recognition device detects that the Y coordinate of the center point representing the first object in the initial detection result is increasing or decreasing, then starting from the first image frame among the N initial image frames, each image frame extracted from the video stream is processed by the Warped Planar Object Detection Network (wpod-NET) model to obtain the corresponding image of the region to be tested.

[0112] In this embodiment of the invention, after the information recognition device detects a triggered event, it performs license plate recognition on each image frame returned by the monitoring camera from the time the event is triggered until its end. This requires the use of license plate detection technology. The information recognition device uses WPOD-NET for license plate detection, which requires first labeling training data. The labeling method is as follows: given an image containing a license plate, label the four vertices of the license plate, starting from the top left corner and proceeding counterclockwise, such as... Figure 4 As shown (the four vertices of the license plate in the image are connected by straight lines). After the annotation is completed, a large amount of labeled data is input into the model for training. The trained model parameters can then be obtained. When an image containing a license plate is input, the model can output the coordinates of the four vertices of the license plate.

[0113] In the embodiments of the present invention, training data is collected by combining manually labeled data with automatically generated data, increasing data diversity while only requiring a small amount of manual collection and labeling. The specific method is to randomly generate license plate images, select a labeled image as the background, and paste the generated license plate onto the license plate position of the image through affine transformation as the generated training sample, making the positions of the 4 vertices of the generated license plate coincide with the positions of the 4 labeled vertices. Since the positions of the 4 vertices of the license plate have been manually labeled before, the positions of these 4 labeled points can be directly used as the labels of the generated image. At the same time, the generated image can also be enhanced by means such as brightness transformation. As[[ID=I]] Figure 5 shown. The license plate information "Lu B 12345" in the image before transformation can be transformed into "Su D 12346" by means of affine transformation.

[0114] S106. Use a preset object detection model to process the image of the待测区域 (to-be-detected area), and obtain multiple characters and corresponding character-related information.

[0115] In the embodiments of the present invention, the information recognition device uses a preset object detection model to process the image of the待测区域 (to-be-detected area), and obtains multiple characters and corresponding character-related information.

[0116] In the embodiments of the present invention, the information recognition device can use the yolov5 model to process the image of the待测区域 (to-be-detected area), and obtain multiple characters and corresponding character-related information. In other embodiments, other models can be used, which are not limited in the embodiments of the present invention.

[0117] In the embodiments of the present invention, after obtaining the image of the待测区域 (to-be-detected area), the information recognition device needs to recognize the license plate characters to determine the license plate number. This solution is based on object detection for license plate character recognition, that is, the object detection method is used to recognize the characters and their positions on the image of the待测区域 (to-be-detected area). This part is trained using Yolov5 on the training data set. Through the trained object detection model, we can obtain the category and position of each character, such as Figure 6As shown in the figure. The object detection model can identify that the image of the area to be measured includes the character "F2242", and obtain the character category F of "F", the position information, and the category probability of 0.91; the character category 2 of "2", the position information, and the category probability of 0.90; the character category 2 of "2", the position information, and the category probability of 0.90; the character category 1 of "4", the position information, and the category probability of 0.90; the character category 2 of "2", the position information, and the category probability of 0.90. This step can also be completed using other object detection algorithms. However, since yolov5 is a relatively mature algorithm at present, both the detection accuracy and the running efficiency can be guaranteed. Therefore, yolov5 is used here. Training the yolov5 model requires a large amount of labeled data. Similar to the license plate detection part, the data collection is difficult and the labeling is cumbersome. In particular, the pictures collected in one area usually only contain the license plates of the local province and the neighboring provinces. Therefore, this solution generates license plate data, pastes it onto the background of the manually collected pictures through affine transformation, and automatically generates the labels of the license plate characters. At the same time, the data augmentation method is used to make the generated license plate characters closer to the actual effect, such as Figure 7 As shown. The new license plate information "Hu PPRBS8X" can be obtained by performing an affine transformation on "Hu 0I0R4H".

[0118] S107. Use the character-related information to perform duplicate removal and optimization processing on multiple characters to obtain the character recognition result composed of characters in a certain order, and stop until the termination detection result represents that the first object is in a state of no displacement change or there is no first object at the termination moment, and obtain multiple character recognition results.

[0119] In the embodiment of the present invention, the information recognition device uses the character-related information to perform duplicate removal and optimization processing on multiple characters to obtain the character recognition result composed of characters in a certain order, and stop until the termination detection result represents that the first object is in a state of no displacement change or there is no first object at the termination moment, and obtain multiple character recognition results. Among them, the termination detection result is obtained by detecting the corresponding N termination image frames.

[0120] In the embodiment of the present invention, the information recognition device uses the character-related information to remove the duplicate characters in multiple characters and剔除 the irrelevant characters outside the license plate area to obtain the character recognition result composed of characters in a certain order. The information recognition device stops until the termination detection result represents that the first object is in a state of no displacement change or there is no first object at the termination moment, and obtains multiple character recognition results.

[0121] In this embodiment of the invention, the information recognition device first determines the state of the vehicle by detecting the vehicle body, and detects the extracted image frames when the vehicle approaches or moves away. Since this solution recognizes the license plate information by capturing the period when the vehicle arrives and leaves, and at the same time uses character-related information to deduplicate and optimize the recognized characters, it overcomes the difficulty of diverse formats in the license plate recognition process, thereby improving the recognition accuracy of license plate information.

[0122] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention. Figure 3 The steps S104 to S107 shown can also be implemented by S108 to S113, which will be explained in conjunction with each step.

[0123] S108. Process the N initial image frames using the first preset detection model to obtain the initial detection results.

[0124] In this embodiment of the invention, the information recognition device uses a first preset detection model to process N initial image frames to obtain initial detection results.

[0125] The first preset detection model can be the Yolov5 model.

[0126] S109. If the initial detection result indicates that at least M of the N initial image frames include the first object, and the distance between the first object and the target area is increasing or decreasing, then each image frame is processed by the second preset detection model to obtain the image of the area to be tested.

[0127] In this embodiment of the invention, if the information recognition device detects that at least M of the N initial image frames contain the first object and the distance between the first object and the target area is increasing or decreasing, then each image frame is processed by the second preset detection model to obtain the image of the area to be tested.

[0128] The second preset detection model can be the WPOD-NET model, and there are no restrictions on the second preset detection model in this embodiment of the invention.

[0129] For example, if the information recognition device detects that at least 15 of the 20 initial image frames represented by the initial detection result include a vehicle, and the distance between the vehicle and the platform is increasing or decreasing, then each image frame is processed by the second preset detection model to obtain the image of the area to be tested.

[0130] S110. The image of the region to be tested is processed using a preset target detection model to obtain multiple characters, as well as the character category probability information and character box information corresponding to each character.

[0131] In this embodiment of the invention, the information recognition device uses a preset target detection model to process the image of the region to be tested, and obtains multiple characters, as well as character category probability information and character frame information corresponding to the multiple characters.

[0132] The preset target detection model can be the YOLOv5 model. In this embodiment of the invention, the type of preset target detection model is not limited.

[0133] The character box information may include: character box area information, character box center point coordinate information, and character box vertex coordinate information.

[0134] S111. Calculate the overlap matrix using information from multiple character boxes, and combine the overlap matrix to remove duplicate characters from multiple characters, thus obtaining multiple first characters.

[0135] In this embodiment of the invention, the information recognition device calculates an overlap matrix using information from multiple character boxes, and performs deduplication on multiple characters using the overlap matrix to obtain multiple first characters.

[0136] In this embodiment of the invention, the information recognition device can obtain multiple overlap degrees by dividing the overlapping area of ​​the character frames of any two characters by the sum of the areas of the character frames of those two characters. The information recognition device can then construct an overlap degree matrix using these multiple overlap degrees in sequence. The information recognition device combines the overlap degree matrix to perform deduplication on multiple characters, resulting in multiple first characters.

[0137] In this embodiment of the invention, the plurality of characters includes T characters; the plurality of character frame area information includes T character frame area information; T is an integer greater than 1. The information recognition device calculates the first group of T overlaps between the first character and each of the T characters based on the character frame area information of the first character to the character frame area information of the Tth character. The information recognition device uses the first group of T overlaps to construct the first row of the overlap matrix, until the Tth character is calculated to have the Tth group of T overlaps with each of the first character to the Tth character, and uses the Tth group of T overlaps to construct the last row of the overlap matrix, thus obtaining the overlap matrix.

[0138] In this embodiment of the invention, the characters output by the preset target detection model may be identified as multiple different classes or multiple identical classes, resulting in a large number of identified characters. In this case, the bounding boxes of these characters have significant overlap. The overlap degree metric is used to measure the overlap range between two character bounding boxes: overlap degree = overlap area / total area. The overlap degree is calculated pairwise for all characters to form an overlap degree matrix D.

[0139] D = [d11, d12, d13, d14, d21, d22, d23, d24, d31, d32, d33, d34, d41, d42, d43, d44]

[0140] Among them, Dij represents the overlap degree between character i and character j. Set a certain threshold thresh, traverse the D matrix. When dij > thresh, it is determined that the overlapping range of these two boxes is relatively large, and one of them is a duplicate recognition. Then, the character with a smaller confidence (confidence represents the probability that this character belongs to the current character category) is removed. Assume confidence(i) < confidence(j), that is, the ith row and the ith column are deleted in D, and then the D matrix is traversed again until all values in the D matrix are less than thresh.

[0141] S112. Combine the first character box information corresponding to multiple first characters to eliminate abnormal characters among the multiple first characters, and obtain multiple second characters.

[0142] In the embodiment of the present invention, the information recognition device combines the first character box information corresponding to multiple first characters to eliminate abnormal characters among the multiple first characters, and obtains multiple second characters.

[0143] In the embodiment of the present invention, the information recognition device combines the center point information of the first character boxes corresponding to multiple first characters, and calculates the density of each first character with other characters. The information recognition device eliminates the character with the smallest density, and obtains multiple second characters.

[0144] S113. Use the second character box information corresponding to multiple second characters to perform hierarchical sorting processing on the multiple second characters to obtain a character recognition result, and stop until the termination detection result indicates that the distance between the first object in at least M of the N termination image frames and the target area has not changed, or there are K termination image frames including the first object in the N start image frames, and obtain multiple character recognition results

[0145] In the embodiment of the present invention, the information recognition device uses the second character box information corresponding to multiple second characters to perform hierarchical sorting processing on the multiple second characters to obtain a character recognition result, and stop until the termination detection result indicates that at least M of the N termination image frames include the first object, and the distance between the first object and the target area has not changed, or there are K termination image frames including the first object in the N start image frames. K is an integer greater than or equal to 1 and less than M.

[0146] In this embodiment of the invention, the information recognition device can obtain more accurate target character recognition results by performing deduplication processing on multiple characters, eliminating abnormal points, and performing hierarchical sorting processing.

[0147] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention. Figure 8 The steps S111 to S112 shown can also be implemented by S114 to S118, which will be explained in conjunction with each step.

[0148] S114. Using the area information of multiple character boxes, calculate the overlap between any two characters, and construct an overlap matrix using the obtained overlap values.

[0149] In this embodiment of the invention, the information recognition device uses the area information of multiple character boxes included in the multiple character box information to calculate the overlap between any two characters, and constructs an overlap matrix through the obtained multiple overlaps.

[0150] S115. Compare multiple overlaps sequentially with preset thresholds to obtain the comparison result for each overlap.

[0151] In this embodiment of the invention, the information recognition device compares multiple overlaps sequentially with a preset threshold to obtain the comparison result for each overlap.

[0152] In this embodiment of the invention, the size of the preset threshold is not limited.

[0153] S116. If the comparison result indicates that the first degree of overlap is greater than the preset threshold, then delete the row and column corresponding to the overlapping character in the overlap matrix to obtain a new overlap matrix, until the overlap in the overlap matrix is ​​less than the preset threshold to obtain the target matrix.

[0154] In this embodiment of the invention, if the information recognition device detects that the first overlap corresponding to the comparison result character is greater than a preset threshold, then the rows and columns corresponding to the overlapping characters in the overlap matrix are deleted to obtain a new overlap matrix, until the overlap in the overlap matrix is ​​less than the preset threshold to obtain the target matrix. The overlapping character is the character with the lowest probability of the character category corresponding to the first overlap.

[0155] For example, combining matrix D = [d11,d12,d13,d14,d21,d22,d23,d24,d31,d32,d33,d34,d41,d42,d43,d44]

[0156] If the comparison result d21 is greater than a preset threshold, d21 represents the overlap between character 2 and character 1. If the character category probability of character 2 is less than the character category probability of character 1, the information recognition device can delete the second row and second column of D.

[0157] S117. Determine multiple first characters by using the characters corresponding to the overlap in any row of the target matrix.

[0158] In this embodiment of the invention, the information recognition device determines multiple first characters by the characters corresponding to the degree of overlap in any row of the target matrix.

[0159] For example, if the target matrix = [d22,d23,d32,d33]

[0160] In this context, the overlap of any row in the target matrix corresponds to characters including characters 2 and 3. Therefore, multiple first characters can include characters 2 and 3.

[0161] S118. The information of multiple first character boxes is processed by a preset anomaly detection model to remove the anomaly character with the lowest density among the multiple first characters, thereby obtaining multiple second characters.

[0162] In this embodiment of the invention, the information recognition device processes multiple character box information through a preset anomaly detection model, removes the anomaly character with the lowest density among multiple first characters, and obtains multiple second characters.

[0163] The preset outlier detection model can be a Local Outlier Factor (LOF) or an outlier detection model. In this embodiment of the invention, the type of preset outlier detection model is not limited.

[0164] like Figure 10 As shown, the preset target detection model incorrectly identified the "1" in the lower left corner. The position of this character is significantly different from other areas in the character set, so the LOF (List of Elements) outlier detection algorithm is used to remove it. The LOF algorithm mainly determines whether a point p is an outlier by comparing the density of each point p with the density of its neighboring points. The lower the density of point p, the more likely it is to be identified as an outlier. The density is calculated based on the distance between points; the farther the distance between points, the lower the density, and the closer the distance, the higher the density.

[0165] In this embodiment of the invention, the information recognition device constructs an overlap matrix by calculating the overlap between any two characters, and then uses the overlap matrix to deduplicate multiple characters. The deduplication effect is good, and the target character recognition result with higher accuracy can be obtained.

[0166] In some embodiments, see Figure 11 , Figure 11 This is a schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention. Figure 8 The shown S113 can also be implemented by S119 to S23, which will be explained in conjunction with each step.

[0167] S119. A fitted line is obtained by linear fitting of multiple center point information in multiple second character box information, and the sum of squared errors of multiple center point information from the fitted line is calculated.

[0168] In this embodiment of the invention, the information recognition device obtains a fitted line by linearly fitting multiple center point information in multiple second character box information, and calculates the sum of squared errors between the multiple center point information and the fitted line.

[0169] S120. Determine the number of character layers in the character recognition result based on the magnitude of the sum of squared errors.

[0170] In this embodiment of the invention, the information recognition device determines the number of character layers in the character recognition result based on the magnitude of the sum of squared errors.

[0171] In this embodiment of the invention, the information recognition device determines that if the sum of squared errors is greater than a second threshold, the number of character layers is two. If the sum of squared errors is not greater than the second threshold, the number of character layers is one. The second threshold can be any real number.

[0172] In this embodiment of the invention, the information recognition device first performs linear fitting on the center coordinates of all characters and calculates the mean square error (MSE) (the sum of squared errors of all points to the fitted straight line). When the MSE exceeds a threshold, it is determined to be a double-layer license plate (because the error would be very small if it were a single-layer plate); otherwise, it is a single-layer license plate. If it is determined to be a double-layer license plate, 5 points are selected from all characters, resulting in 21 combinations. Linear fitting is performed on each combination to calculate the MSE, and the character combination with the smallest MSE is retained. Because the 5 characters of the lower layer license plate are already on a straight line, the error is definitely the smallest among all combinations. Figure 6 As shown, the combination of characters "F2242" indicates a single-layer license plate.

[0173] S121. If the character layer is two, then extract multiple character combinations including any P second characters from multiple second characters.

[0174] In this embodiment of the invention, if the number of character layers is two, the information recognition device extracts multiple character combinations, including any P second characters, from multiple second characters. P is an integer greater than 1 and less than T.

[0175] In this embodiment of the invention, if the number of character layers is one, the information recognition device concatenates the plurality of second characters according to the size order of their respective corresponding horizontal coordinate information to obtain the character recognition result.

[0176] S122. Perform linear fitting on multiple character combinations and calculate multiple sums of squared errors. Determine the P third characters included in the character combination corresponding to the smallest sum of squared errors.

[0177] In this embodiment of the invention, the information recognition device performs linear fitting on multiple character combinations and calculates multiple sums of squared errors, determining the P third characters included in the character combination corresponding to the smallest sum of squared errors.

[0178] In this embodiment of the invention, the information recognition device performs linear fitting on the center point coordinates of the characters within multiple character combinations to obtain multiple sets of fitted lines. The information recognition device then calculates the sum of squared errors for each group of characters, resulting in multiple sums of squared errors.

[0179] S123. Concatenate the remaining characters according to the order of their corresponding horizontal coordinates in the second character box, and then concatenate the P third characters according to the order of their corresponding horizontal coordinates to obtain the character recognition result.

[0180] In this embodiment of the invention, the information recognition device concatenates the remaining characters according to the order of their corresponding horizontal coordinates in the second character box information, and then concatenates the P third characters according to the order of their corresponding horizontal coordinates to obtain the character recognition result. The remaining characters are the characters other than the P third characters among the multiple second characters.

[0181] In this embodiment of the invention, the information recognition device performs linear fitting on multiple second characters, and determines the number of layers of a character by the sum of squared errors between each character and the fitted line. Since the number of layers of license plate characters in real-world scenarios is uncertain, this method can more accurately determine the number of layers, thus obtaining accurate target character recognition results.

[0182] In some embodiments, see Figure 12 , Figure 12 This is a schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention. Figure 1 The shown S101 can also be implemented by S124 to S26, which will be explained in conjunction with each step.

[0183] S124. Extract the current image frame corresponding to the current moment from the video stream.

[0184] In this embodiment of the invention, the information recognition device extracts the current image frame corresponding to the current moment from the video stream.

[0185] For example, a camera captures a video stream of the current predetermined area in real time and sends it to an information recognition device. Upon receiving the real-time video stream, the information recognition device extracts the image frame corresponding to the current moment.

[0186] S125. Starting from the current image frame in the video stream, extract one image frame every predetermined duration or predetermined number of image frames along the time axis until N-1 image frames are extracted.

[0187] In this embodiment of the invention, the information recognition device extracts an image frame from the video stream starting from the current image frame and along the time axis every predetermined duration or predetermined number of image frames until N-1 image frames are extracted.

[0188] In this embodiment of the invention, the video stream includes multiple image frames distributed along the time axis. The information recognition device can extract one image frame every second, starting from the current image frame, until N-1 image frames have been extracted. In this embodiment of the invention, there is no limit to the predetermined duration.

[0189] In this embodiment of the invention, the video stream includes multiple image frames distributed along the time axis. The information recognition device can extract one image frame every 5 image frames, starting from the current image frame, until N-1 image frames are extracted. In this embodiment of the invention, there is no limit to the predetermined number.

[0190] S126. Combine the current image frame and N-1 image frames along the time axis to obtain N initial image frames.

[0191] In this embodiment of the invention, the information recognition device combines the current image frame and N-1 image frames along the time axis to obtain N initial image frames.

[0192] In this embodiment of the invention, the information recognition device combines the current image frame and N-1 image frames in order from front to back along their respective time points to obtain N image frames.

[0193] In this embodiment of the invention, the information recognition device extracts N image frames at the current moment based on the sliding window method. By utilizing the dynamic features of the N image frames, it can accurately determine whether a vehicle has arrived at or left the platform.

[0194] In some embodiments, see Figure 12 , Figure 12 This is a schematic diagram of an optional process for the information recognition method provided in an embodiment of the present invention. Figure 1 The shown S103 can also be implemented by S127 to S28, which will be explained in conjunction with each step.

[0195] S127. Add up the character category probability information corresponding to each character in the multiple character recognition results to obtain multiple total probabilities.

[0196] In this embodiment of the invention, the information recognition device adds up the character category probability information corresponding to each character in the multiple character recognition results to obtain multiple total probabilities.

[0197] S128. Determine the character recognition result corresponding to the largest total probability among multiple total probabilities as the target character recognition result.

[0198] In this embodiment of the invention, the information recognition device determines the character recognition result corresponding to the largest total probability among multiple total probabilities as the target character recognition result.

[0199] In this embodiment of the invention, the information recognition device determines which characters belong to the upper layer and which belong to the lower layer (all characters on a single-layer license plate are considered as the lower layer license plate). Because the relative positions of license plates on the same layer are fixed on the x-axis, the characters of the upper layer license plate are first concatenated in ascending order of x-coordinate. Then, the characters of the lower layer license plate are concatenated with the characters of the upper layer license plate in ascending order of x-coordinate. The final string obtained is the final license plate number. Since license plate recognition begins when an event is triggered and ends after the event, there will be multiple recognition results for a single vehicle. Due to environmental and other factors, the recognition results may be inconsistent each time. Therefore, it is necessary to merge the multiple recognition results. The merging method is to add the probabilities of each character in each recognition result for each position, and take the character with the highest overall probability as the character corresponding to that position.

[0200] In this embodiment of the invention, the information recognition device determines the target character recognition result with the highest probability among multiple character recognition results, thus obtaining a more accurate recognition result.

[0201] In some embodiments, see Figure 13 , Figure 13 This is an optional flowchart illustrating the information recognition method provided in an embodiment of the present invention, which will be explained in conjunction with each step.

[0202] S201, Mark the rear compartment data.

[0203] In this embodiment of the invention, the target object needs to annotate the rear compartment in a large number of images to form sample data for training.

[0204] S202, Training vehicle body detection model.

[0205] S203, Label the license plate and license plate character data.

[0206] S204. Generate license plate numbers, license plate characters, and their marking data.

[0207] S205, Training license plate and license plate character detection model.

[0208] The system flow of this solution includes a training process and a usage process. In the training process, it is necessary to collect and label data on the vehicle body, license plate, and license plate characters, while automatically generating highly realistic license plate and license plate character data to train the detection model for the vehicle body, license plate, and license plate characters. After the model training is complete, it is deployed on an edge computing box for practical use.

[0209] S206. Extract images from the video stream in real time (one frame every 0.2 seconds).

[0210] The information recognition device extracts images from the video stream of the surveillance camera in real time, extracting one frame every 0.2 seconds.

[0211] S207, Vehicle inspection.

[0212] The image extracted by the information recognition device is used by the vehicle recognition module to determine whether there is a vehicle on the current platform.

[0213] S208, Vehicle Dynamics / Platform Status / Event Recognition.

[0214] The information recognition device uses the vehicle recognition status of the current frame and the recognition status of the previous N frames to jointly determine the current status of the vehicle and the platform, as well as whether an event has occurred.

[0215] S209, License Plate Inspection.

[0216] If event monitoring is triggered, the information recognition device will perform license plate detection; detection will stop after the event ends.

[0217] S210, License Plate Character Detection.

[0218] The information recognition device expands the detected license plate area outward by a certain range, captures it, and sends it to the license plate recognition module.

[0219] S211, Generate license plate number.

[0220] The information recognition device outputs the license plate recognition result.

[0221] During deployment, each deep learning model is converted to TensorRT format to optimize resource usage. In actual use, the surveillance cameras are placed at the top of the warehouse door, shooting at an angle downwards to ensure that the platform monitoring footage is collected without affecting production. The cameras are directly connected to the edge box, where all vehicle / license plate recognition modules are deployed. The video collected by the cameras is analyzed, and the recognition results are reported to the online system. Due to the large area captured by the cameras, this solution only monitors the corresponding parking spaces on the platform to eliminate interference from other areas. One camera can be configured with multiple monitoring areas, each independent of the others, performing vehicle / license plate recognition separately. The overall process is as follows: Figure 14 As shown, camera 100 collects video streams of the monitored area and transmits them to edge box 101. Edge box 101 processes N image frames at each moment in the video stream, triggers event monitoring, and finally outputs the license plate number.

[0222] See Figure 15 , Figure 15 This is a schematic diagram of the structure of the information recognition device provided in an embodiment of the present invention.

[0223] This invention also provides an information identification device 800, including: an acquisition and extraction unit 803, a detection and processing unit 804, and a determination unit 805.

[0224] The acquisition and extraction unit is used to acquire video streams of a predetermined area in real time, and extract N starting image frames corresponding to the current moment from the video stream based on a sliding window method; N is an integer greater than 1.

[0225] The detection processing unit is configured to, if a first object is detected as a starting state in any of the N initial image frames, perform deduplication optimization processing on the characters in the test region image of each image frame to obtain a character recognition result, until the first object is detected as a ending state in any of the N ending image frames, thereby obtaining multiple character recognition results; wherein, the N ending image frames are extracted from the video stream at an ending time after any time interval from the current time; each image frame belongs to multiple image frames extracted from the video stream from the current time to the ending time;

[0226] The determining unit is used to determine the target character recognition result based on the multiple character recognition results.

[0227] In this embodiment of the invention, the detection processing unit 804 in the information recognition device 800 is used to detect the N initial image frames to obtain an initial detection result; if the initial detection result indicates that the first object is close to or far from the target area, then starting from the first image frame among the N initial image frames, each image frame extracted from the video stream is processed to obtain the corresponding image of the area to be tested; the image of the area to be tested is processed using a preset target detection model to obtain multiple characters and corresponding character-related information; the multiple characters are deduplicated and optimized using the character-related information to obtain the character recognition result composed of characters in a certain order, until the termination detection result indicates that the first object is in a state of no displacement change or that there is no first object at the termination time, and the multiple character recognition result is obtained; the termination detection result is obtained by detecting the corresponding N termination image frames.

[0228] In this embodiment of the invention, the detection processing unit 804 in the information recognition device 800 is used to process the N initial image frames using a first preset detection model to obtain the initial detection result; if the initial detection result indicates that at least M of the N initial image frames include the first object, and the distance between the first object and the target area is increasing or decreasing, then each image frame is processed by a second preset detection model to obtain the image of the area to be tested; M is an integer greater than 1 and less than N.

[0229] In this embodiment of the invention, the detection processing unit 804 in the information recognition device 800 is used to process the image of the region to be tested using a preset target detection model to obtain the plurality of characters, and the character category probability information and character box information corresponding to the plurality of characters respectively.

[0230] In this embodiment of the invention, the detection processing unit 804 in the information recognition device 800 is used to calculate an overlap matrix using multiple character frame information, and to perform deduplication processing on the multiple characters in combination with the overlap matrix to obtain multiple first characters; to remove abnormal characters from the multiple first characters by combining the first character frame information corresponding to each of the multiple first characters to obtain multiple second characters; to perform hierarchical sorting processing on the multiple second characters using the second character frame information corresponding to each of the multiple second characters to obtain the character recognition result, until the termination detection result indicates that the distance between the first object and the target region has not changed in at least M of the N termination image frames, or the first object is included in K of the N starting image frames, and the multiple character recognition result is obtained; K is an integer greater than or equal to 1 and less than M.

[0231] In this embodiment of the invention, the detection processing unit 804 in the information recognition device 800 is used to calculate the overlap between any two characters using the area information of the multiple character boxes included in the multiple character box information, and to construct the overlap matrix using the obtained multiple overlaps; the multiple overlaps are compared sequentially with a preset threshold to obtain a comparison result for each overlap; if the comparison result indicates that the corresponding first overlap is greater than the preset threshold, the row and column corresponding to the overlapping character in the overlap matrix are deleted to obtain a new overlap matrix, until the overlap in the overlap matrix is ​​less than the preset threshold to obtain a target matrix; the overlapping character is the character with the lowest probability of the character category corresponding to the first overlap; the multiple first characters are determined by the characters corresponding to the overlap in any row of the target matrix.

[0232] In this embodiment of the invention, the plurality of characters includes: T characters; the plurality of character frame area information includes: T character frame area information; T is an integer greater than 1; the detection processing unit 804 in the information recognition device 800 is used to calculate the overlap between any two characters using the plurality of character frame area information included in the plurality of character frame information, and to construct the overlap matrix using the obtained plurality of overlaps, including: calculating the first group of T overlaps between the first character and the characters from the first character to the Tth character based on the character frame area information of the first character to the character frame area information of the Tth character; constructing the first row of the overlap matrix using the first group of T overlaps, until the Tth group of T overlaps between the Tth character and the characters from the first character to the Tth character is calculated, and constructing the last row of the overlap matrix using the Tth group of T overlaps, thereby obtaining the overlap matrix.

[0233] In this embodiment of the invention, the detection processing unit 804 in the information recognition device 800 is used to process multiple first character box information through a preset anomaly detection model, remove the abnormal character with the lowest density among the multiple first characters, and obtain the multiple second characters.

[0234] In this embodiment of the invention, the detection processing unit 804 in the information recognition device 800 is used to obtain a fitting line by linear fitting through multiple center point information in the multiple second character box information, and calculate the sum of squared errors between the multiple center point information and the fitting line; determine the number of character layers of the character recognition result according to the magnitude of the sum of squared errors; if the number of character layers is two, extract multiple character combinations including any P second characters from the multiple second characters; P is an integer greater than 1 and less than T; perform linear fitting on the multiple character combinations respectively, and calculate multiple sums of squared errors, and determine the P third characters included in the character combination corresponding to the smallest sum of squared errors; concatenate the remaining characters according to the size order of the horizontal coordinate information in their respective second character box information, and then concatenate the P third characters according to the size order of their respective horizontal coordinate information to obtain the character recognition result; the remaining characters are the characters other than the P third characters among the multiple second characters.

[0235] In this embodiment of the invention, the detection and processing unit 804 in the information recognition device 800 is used to concatenate the plurality of second characters according to the size order of their respective corresponding horizontal coordinate information if the number of character layers is one layer, so as to obtain the character recognition result.

[0236] In this embodiment of the invention, the determining unit 805 in the information recognition device 800 is used to add the character category probability information corresponding to each character in the plurality of character recognition results to obtain a plurality of total probabilities; and determine the character recognition result corresponding to the largest total probability among the plurality of total probabilities as the target character recognition result.

[0237] In this embodiment of the invention, the acquisition and extraction unit 803 in the information recognition device 800 is used to extract the current image frame corresponding to the current moment in the video stream; starting from the current image frame, extracting one image frame along the time axis every predetermined duration or predetermined number of image frames until N-1 image frames are extracted; combining the current image frame and the N-1 image frames along the time axis to obtain the N initial image frames.

[0238] In this embodiment of the invention, the acquisition and extraction unit 803 in the information recognition device 800 is used to acquire video streams of a predetermined area in real time, and extract N starting image frames corresponding to the current time in the video stream based on a sliding window method; N is an integer greater than 1; if the detection and processing unit 804 detects that the first object in the N starting image frames is in the initial state, it performs deduplication and optimization processing on the characters in the image of the region to be tested in each image frame to obtain character recognition results, until the first object in the N ending image frames is detected to be in the ending state, thus obtaining multiple character recognition results; wherein, the N ending image frames are extracted from the video stream at the ending time after any time interval from the current time; the image frames belong to multiple image frames extracted from the video stream from the current time to the ending time; the determination unit 805 determines the target character recognition result based on the multiple character recognition results. Since this solution recognizes license plate information by capturing the period between the arrival and departure of the first object, and simultaneously performs deduplication and optimization processing on the recognized characters, it overcomes the difficulty of diverse formats in the license plate recognition process, thereby improving the recognition accuracy of license plate information.

[0239] It should be noted that, in the embodiments of the present invention, if the above-described information identification method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an information identification device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiments of the present invention are not limited to any specific hardware and software combination.

[0240] Correspondingly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0241] Correspondingly, this embodiment of the invention provides an information identification device, including a memory 802 and a processor 801. The memory 802 stores a computer program that can run on the processor 801. When the processor 801 executes the program, it implements the steps in the above method.

[0242] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of the present invention, please refer to the descriptions of the method embodiments of the present invention for understanding.

[0243] It should be noted that, Figure 16 This is a schematic diagram of a hardware entity of the information recognition device provided in an embodiment of the present invention, such as... Figure 16 As shown, the hardware entity of the information recognition device 800 includes: a processor 801 and a memory 802, wherein;

[0244] The processor 801 typically controls the overall operation of the information identification device 800.

[0245] The memory 802 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or already processed by the processor 801 and the various modules in the information recognition device 800 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0246] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of the invention, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the invention. The sequence numbers of the above-described embodiments of the invention are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0247] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0248] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0249] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0250] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0251] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0252] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0253] The above description is merely an embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An information recognition method characterized by comprising: The method comprises the following steps: real-time acquisition of a video stream of a predetermined area, and extraction of N starting image frames corresponding to a current time from the video stream based on a sliding window method; N is an integer greater than 1; if it is detected that a first object in the N starting image frames is in a starting state, then a de-duplication optimization process is performed on characters in a to-be-tested area image in each image frame to obtain a character recognition result, until it is detected that the first object in N terminal image frames is in a terminal state, and a plurality of character recognition results are obtained; wherein the N terminal image frames are extracted at a terminal time after any time length from the current time in the video stream; each image frame belongs to a plurality of image frames extracted in the video stream from the current time to the terminal time; an overlap matrix is calculated using a plurality of character box information, and the plurality of characters are de-duplicated based on the overlap matrix to obtain a plurality of first characters; abnormal characters in the plurality of first characters are removed based on first character box information corresponding to the plurality of first characters to obtain a plurality of second characters; the plurality of second characters are sorted in layers based on second character box information corresponding to the plurality of second characters to obtain the character recognition result, until a terminal detection result represents that the distance between the first object and a target area in at least M terminal image frames of the N terminal image frames does not change, or the first object is included in K terminal image frames of the N starting image frames, and the plurality of character recognition results are obtained; K is an integer greater than or equal to 1 and less than M; based on the plurality of character recognition results, a target character recognition result is determined.

2. The information recognition method according to claim 1, characterized by, If it is detected that a first object in the N starting image frames is in a starting state, then a de-duplication optimization process is performed on characters in a to-be-tested area image in each image frame to obtain a character recognition result, until it is detected that the first object in N terminal image frames is in a terminal state, and a plurality of character recognition results are obtained, including: detecting the N starting image frames to obtain a starting detection result; if the starting detection result represents that the first object is in a state of approaching or moving away from a target area, then a first image frame in the N starting image frames is taken as a starting point to process each image frame extracted in the video stream to obtain a corresponding to-be-tested area image; processing the to-be-tested area image using a preset target detection model to obtain a plurality of characters and corresponding character-related information; performing a de-duplication optimization process on the plurality of characters using the character-related information to obtain the character recognition result composed of characters in a certain order, until a terminal detection result represents that the first object is in a state of no displacement change or the first object is in a state of no presence at the terminal time, and the plurality of character recognition results are obtained; the terminal detection result is obtained by detecting the N terminal image frames.

3. The information recognition method according to claim 2, characterized by, The detection of the N starting image frames to obtain a starting detection result comprises: processing the N starting image frames by using a first preset detection model to obtain the starting detection result; if the starting detection result represents that the first object is in a state of approaching or moving away from the target region, processing each image frame extracted from the video stream starting from a first image frame in the N starting image frames to obtain a corresponding to-be-detected region image, including: if the starting detection result represents that the first object is included in at least M starting image frames in the N starting image frames, and the distance between the first object and the target region is increasing or decreasing, processing each image frame by using a second preset detection model to obtain the to-be-detected region image; M is an integer greater than 1 and less than N.

4. The information recognition method according to claim 2, characterized by, processing the to-be-detected region image by using a preset target detection model to obtain a plurality of characters and corresponding character-related information, including: processing the to-be-detected region image by using a preset target detection model to obtain the plurality of characters and character class probability information and character box information corresponding to the plurality of characters, respectively.

5. The information recognition method according to claim 1, characterized by, calculating an overlap matrix by using a plurality of character box information, and performing deduplication processing on the plurality of characters based on the overlap matrix to obtain a plurality of first characters, including: calculating an overlap between any two characters by using a plurality of character box area information included in the plurality of character box information, and constructing the overlap matrix by using a plurality of overlaps obtained; comparing the plurality of overlaps with a preset threshold value in sequence to obtain a comparison result of each overlap; if the comparison result represents that a first overlap corresponding to the comparison result is greater than the preset threshold value, deleting a row and a column corresponding to an overlapping character in the overlap matrix to obtain a new overlap matrix until the overlaps in the overlap matrix are all less than the preset threshold value to obtain a target matrix; the overlapping character is a character with the smallest character class probability corresponding to the first overlap; determining the plurality of first characters by using characters corresponding to the overlaps in any row included in the target matrix.

6. The information recognition method according to claim 5, characterized by, The plurality of characters include T characters, and the plurality of character box area information includes T character box area information; T is an integer greater than 1. calculating an overlap between any two characters by using a plurality of character box area information included in the plurality of character box information, and constructing the overlap matrix by using a plurality of overlaps obtained, including: calculating a first group of T overlaps between the first character and the first character to the Tth character, respectively, according to the character box area information of the first character to the character box area information of the Tth character; constructing a first row of the overlap matrix by using the first group of T overlaps until a Tth group of T overlaps between the Tth character and the first character to the Tth character, respectively, are calculated, constructing a last row of the overlap matrix by using the Tth group of T overlaps, and further obtaining the overlap matrix.

7. The information recognition method according to Claim 1, characterized by, eliminating abnormal characters in the plurality of first characters based on first character box information corresponding to the plurality of first characters to obtain a plurality of second characters, including: The first character frame information is processed by a preset anomaly point detection model, and the anomaly character with the lowest density between other first characters is removed from the plurality of first characters to obtain a plurality of second characters.

8. The information recognition method according to claim 1, characterized by, The second character frame information corresponding to the plurality of second characters is used to perform hierarchical sorting processing on the plurality of second characters to obtain the character recognition result, including: A fitting line is obtained by linear fitting of the plurality of center point information in the plurality of second character frame information, and an error sum of squares of the plurality of center point information from the fitting line is calculated; The character layer number of the character recognition result is determined according to the size of the error sum of squares; If the character layer number is two, a plurality of character combinations including any P second characters are extracted from the plurality of second characters; P is an integer greater than 1 and less than T; The plurality of character combinations are linearly fitted respectively, and a plurality of error sums of squares are calculated to determine the P third characters included in the character combination corresponding to the smallest error sum of squares; The remaining characters are spliced according to the size order of the horizontal coordinate information in the respective corresponding second character frame information, and the P third characters are spliced according to the size order of the respective corresponding horizontal coordinate information to obtain the character recognition result; the remaining characters are characters in the plurality of second characters except the P third characters.

9. The information recognition method according to claim 8, characterized by, After the character layer number of the character recognition result is determined according to the size of the error sum of squares, the method further includes: If the character layer number is one, the plurality of second characters are spliced according to the size order of the respective corresponding horizontal coordinate information to obtain the character recognition result.

10. The information recognition method according to Claim 1, characterized by, The target character recognition result is determined based on the plurality of character recognition results, including: The character category probability information corresponding to each character in the plurality of character recognition results is added to obtain a plurality of total probabilities; The character recognition result corresponding to the maximum total probability in the plurality of total probabilities is determined as the target character recognition result.

11. The information recognition method according to Claim 1, characterized by, The method based on the sliding window extracts N starting image frames corresponding to the current time from the video stream, including: A current image frame corresponding to the current time is extracted from the video stream; Starting from the current image frame, an image frame is extracted every predetermined time length or predetermined number of image frames along the time axis until N-1 image frames are extracted; The current image frame and the N-1 image frames are combined along the time axis to obtain the N starting image frames.

12. An information recognizing apparatus characterized by comprising: Including: The acquisition and extraction unit is configured to acquire a video stream of a predetermined area in real time, and extract N starting image frames corresponding to the current time from the video stream based on the sliding window method; N is an integer greater than 1. The detection processing unit is configured to perform de-duplication optimization on the characters in the to-be-detected region of each image frame to obtain character recognition results, until the first object in the N end image frames is detected to be in the end state, and a plurality of character recognition results are obtained. The detection processing unit is configured to calculate an overlap matrix using the plurality of character box information, perform de-duplication on the plurality of characters based on the overlap matrix, obtain a plurality of first characters, remove abnormal characters in the plurality of first characters based on the first character box information corresponding to the plurality of first characters, obtain a plurality of second characters, perform hierarchical sorting on the plurality of second characters based on the second character box information corresponding to the plurality of second characters, and obtain the character recognition results until the termination detection result indicates that the distance between the first object and the target region in at least M end image frames of the N end image frames does not change, or the first object is included in K end image frames of the N start image frames, and the plurality of character recognition results are obtained. K is an integer greater than or equal to 1 and less than M. The determination unit is configured to determine a target character recognition result based on the plurality of character recognition results.

13. An information recognizing apparatus characterized by comprising: The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 11.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 11.

Images