Image processing method and device

Through automatic screening and batch labeling of image data, the problem of low image data preprocessing efficiency in traffic management systems is solved, the evaluation efficiency and accuracy of image recognition algorithms are improved, and efficient data preparation and recognition results analysis are achieved.

CN111476107BActive Publication Date: 2024-11-08PINGAN INT SMART CITY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010190279.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-03-18
Publication Date
2024-11-08
Estimated Expiration
2040-03-18

AI Technical Summary

Technical Problem

In the traffic management system, it is difficult for the prior art to efficiently preprocess image data, resulting in inefficient evaluation of image recognition algorithms, and manual labeling is prone to errors, affecting the recognition accuracy.

Method used

By determining the image collection in the same time period under the same scene, filtering out pictures with the same image size and resolution, automatically labeling traffic markers and vehicle position changes, generating batch marking information, and using adaptive algorithms to match the marking information to all pictures, simplifying Manual operation and improve data preparation efficiency.

Benefits of technology

It greatly shortens the preparation time of test data, improves testing efficiency, reduces manual operation errors, improves the accuracy of the algorithm and product recognition accuracy, and generates test reports that are easy to analyze.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111476107B_ABST
    Figure CN111476107B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses an image processing method, including: determining a first image set, the first image set includes multiple first images, the multiple first images are images taken of a target vehicle in the same scene and the same time period; determining one or more second image sets from the first image set, each of the one or more second image sets includes multiple second images, and each second image set records a pending illegal driving process of the target vehicle; marking the reference position of the traffic sign in at least one of the one or more second image sets, and respectively identifying the position change of the target vehicle in the one or more second image sets to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal. By adopting the embodiment of the present invention, the efficiency of test data preprocessing can be improved and the algorithm can be effectively evaluated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to an image processing method and device. Background Art

[0002] In the current traffic management system, image recognition algorithms are usually used to identify multiple images taken by cameras to determine whether the driving behavior of the corresponding vehicle in the image is illegal or illegal. Therefore, it is crucial to select a suitable recognition algorithm from a large number of image recognition algorithms to accurately identify and judge traffic violations. When selecting an algorithm, it is necessary to effectively evaluate the precision and accuracy of each algorithm based on a unified algorithm accuracy standard, so as to determine the recognition match between the algorithm and the corresponding scene.

[0003] In the process of evaluating a specific recognition algorithm, first of all, a large amount of test data (such as image data) needs to be cleaned, screened, and classified before testing, and this step takes a certain amount of time; then the data that has undergone the aforementioned preprocessing needs to be cropped and the relevant information labeled, but manual operations are time-consuming and may result in errors, resulting in inaccurate labeling of reference objects in the image and other problems.

[0004] Therefore, how to improve the efficiency of data preprocessing before testing, so as to effectively evaluate the image recognition algorithm, is a problem that needs to be solved. Summary of the invention

[0005] The embodiments of the present invention provide an image processing method and device, which can improve the efficiency of data preprocessing before testing, thereby effectively evaluating an image recognition algorithm.

[0006] In a first aspect, an embodiment of the present invention provides an image processing method, which may include:

[0007] Determine a first image set, where the first image set includes a plurality of first images, where the plurality of first images are images taken of a target vehicle in a same scene and within a same time period;

[0008] Determine one or more second image sets from the first image set, each of the one or more second image sets includes a plurality of second images, and each of the second image sets records a pending illegal driving process of the target vehicle;

[0009] The reference position of the traffic sign in at least one of the one or more second image sets is marked, and the position change of the target vehicle in the one or more second image sets is identified respectively, so as to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal.

[0010] In the implementation of the embodiment of the present invention, multiple images with the same image size and image resolution are selected from the image data according to the image size and image resolution; wherein the image size taken by the same camera (i.e., the bayonet) in a specific scene is the same, and the resolution at a specific moment is almost the same. Then, the multiple first images selected can be considered to be taken by the same camera. According to the shooting rules of the second image (for example, the camera takes a second image before the illegal driving occurs; takes a second image during the illegal driving process; and takes a second image after the illegal driving process ends), one or more groups of second images are selected from the multiple first images. Each group of second images fully records an illegal driving process. Select a picture from any group of second pictures, and mark the reference objects in the picture related to the illegal driving process. More specifically, the remaining pictures are batch marked according to the already marked pictures. Or all the reference objects contained in all pictures are marked at one time according to the marking rules. Specifically, by marking the positions of reference objects related to illegal standards in the image, such as the position of traffic lights, the position of stop lines, the position of solid white lines, etc., the relevant annotation information is automatically generated, and according to the relationship between the image resolution, the annotation information is adaptively matched to all the images above through a packaged adaptive algorithm, thereby completing the batch annotation operation. The test data is preprocessed by the program, and the present invention can greatly shorten the preparation and preprocessing time of the test data and improve the test efficiency. Through the script that batch reads the annotation information, the test data is transferred to the tested algorithm in batches, which simplifies the manual operation steps of the tester and avoids errors. Through the automatic annotation and statistical method of the script, a standard test report is automatically generated, the test results are recorded, and an algorithm confusion matrix that is easy to analyze is generated, and the test results are classified according to the needs, which is convenient for the algorithm team to analyze the results of the tested pictures, improve the algorithm accuracy, and optimize the product recognition accuracy.

[0011] In a possible implementation manner, each of the plurality of first images includes an identifier indicating the first image; and determining the first image set includes: determining the first image set according to the identifier of the first image.

[0012] In a possible implementation manner, the image size and image resolution of each of the multiple first images are the same.

[0013] In a possible implementation, marking a reference position of a traffic sign in at least one of the one or more second image sets, and respectively identifying a position change of the target vehicle in the one or more second image sets to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal, includes:

[0014] marking a reference position of a traffic sign in a target second image set in the one or more second image sets;

[0015] The reference positions of the traffic signs in the second image sets other than the target second image set in the one or more second image sets are marked according to the reference positions of the traffic signs in the target second image set and a position marking algorithm.

[0016] In one possible implementation, the respectively identifying the position changes of the target vehicle in the one or more second image sets to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal includes: respectively identifying the position changes of the target vehicle in the one or more second image sets through a traffic recognition algorithm to obtain the judgment results of the pending illegal driving corresponding to the one or more second image sets; the judgment results include illegal driving and non-illegal driving.

[0017] In a possible implementation, the method further includes:

[0018] Comparing the determination results of the pending illegal driving corresponding to the one or more second image sets respectively with the reference results corresponding to the one or more second image sets respectively;

[0019] Evaluate the accuracy of the traffic identification algorithm.

[0020] In a possible implementation, the evaluating the accuracy of the traffic recognition algorithm includes:

[0021] The accuracy of the traffic recognition algorithm is evaluated by precision, recall and balanced F-score.

[0022] In a possible implementation, a plurality of third pictures are obtained, wherein the image size and image resolution of each of the plurality of third pictures are the same or different; and from the plurality of third pictures, the plurality of first pictures having the same image size and image resolution are determined.

[0023] In a possible implementation, the shooting rule of the second picture is to shoot one or more second pictures before the illegal driving process occurs; shoot one or more second pictures when the illegal driving process occurs; shoot one or more second pictures after the illegal driving process occurs.

[0024] In a possible implementation, the traffic sign includes a traffic light and / or a traffic line. For example, a traffic light may include a red light; and a traffic line may include a stop line and a solid white line.

[0025] In a second aspect, an embodiment of the present invention provides an image processing device, which may include:

[0026] A determination unit, configured to determine a first image set, wherein the first image set includes a plurality of first images, and the plurality of first images are images taken of a target vehicle in a same scene and within a same time period;

[0027] A screening unit, configured to determine one or more second image sets from the first image set, each of the one or more second image sets comprising a plurality of second images, and each of the second image sets recording a pending illegal driving process of the target vehicle;

[0028] A labeling unit is used to label the reference position of the traffic sign in at least one of the one or more second image sets, and to identify the position change of the target vehicle in the one or more second image sets, so as to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal.

[0029] In a possible implementation manner, each of the multiple first images includes an identifier indicating the first image; and the determination unit is specifically configured to determine the first image set according to the identifier of the first image.

[0030] In a possible implementation manner, the image size and image resolution of each of the multiple first images are the same.

[0031] In a possible implementation, the labeling unit is specifically used to:

[0032] marking a reference position of a traffic sign in a target second image set in the one or more second image sets;

[0033] The reference positions of the traffic signs in the second image sets other than the target second image set in the one or more second image sets are marked according to the reference positions of the traffic signs in the target second image set and a position marking algorithm.

[0034] In a possible implementation, the labeling unit is specifically used to: identify the position changes of the target vehicle in the one or more second image sets respectively through a traffic recognition algorithm, and obtain the judgment results of the pending illegal driving corresponding to the one or more second image sets respectively; the judgment results include illegal driving and non-illegal driving.

[0035] In a possible implementation manner, the device further includes an evaluation unit, configured to:

[0036] Comparing the determination results of the pending illegal driving corresponding to the one or more second image sets respectively with the reference results corresponding to the one or more second image sets respectively;

[0037] Evaluate the accuracy of the traffic identification algorithm.

[0038] In a possible implementation, the evaluation unit is specifically configured to:

[0039] The accuracy of the traffic recognition algorithm is evaluated by precision, recall and balanced F-score.

[0040] In a third aspect, an embodiment of the present invention provides an image processing device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method described in the first aspect.

[0041] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method described in the first aspect.

[0042] In a fifth aspect, an embodiment of the present invention provides a computer program, which includes instructions. When the computer program is executed by a computer, the computer can execute part or all of the steps of any one of the image processing methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0044] Figure 1 is a schematic diagram of an image processing system architecture provided by an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present invention;

[0046] Figure 3 is a flowchart of an image processing method provided by an embodiment of the present invention;

[0047] Figure 4 is a structural schematic diagram of an image processing device provided by an embodiment of the present invention;

[0048] Figure 5 It is a structural schematic diagram of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present invention provide an image processing method and device, which can quickly and effectively identify the text meaning of a text carrier in a picture to be identified.

[0050] The terms "including" and "having" and any variations thereof that appear in the description, claims and drawings of the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices. In addition, the terms "first", "second" and "third" etc. are used to distinguish different objects, rather than to describe a specific order. The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention.

[0051] First, some terms in the embodiments of the present invention are explained to facilitate understanding by those skilled in the art.

[0052] (1) Security checkpoint is the abbreviation of road traffic security checkpoint monitoring system, which refers to a road traffic on-site monitoring system that relies on specific places on the road, such as toll booths, traffic or security checkpoints, etc., to photograph, record and process all motor vehicles passing through the checkpoint.

[0053] (2) The traditional image recognition process is divided into four steps: image acquisition → image preprocessing → feature extraction → image recognition. Image recognition may be based on the main features of the image. Each image has its own features, such as the letter A has a point, P has a circle, and the center of the Y has an acute angle.

[0054] (3) Precision is an indicator to measure the signal-to-noise ratio of a retrieval system, that is, the percentage of relevant documents retrieved to all documents retrieved. It is generally expressed as: Precision = (the amount of relevant information retrieved / the total amount of information retrieved) x 100%.

[0055] (4) Recall Ratio refers to the ratio of the amount of relevant information retrieved from the database to the total amount. The absolute value of the recall ratio can be estimated based on the content and quantity of the database.

[0056] (5) F1 score is an indicator used in statistics to measure the accuracy of a binary classification model. It takes into account both the precision and recall of the classification model. The F1 score can be regarded as a harmonic mean of the model's precision and recall. Its maximum value is 1 and its minimum value is 0.

[0057] (6) Traffic signals can be divided into traffic lights, traffic signs, traffic markings, etc.

[0058] (7) Hypertext Transfer Protocol request (http request) refers to the request message from the client to the server. It includes: the resource request method, resource identifier and protocol used in the first line of the message.

[0059] (8) Python is a cross-platform computer programming language. It is an object-oriented, dynamically typed language that was originally designed for writing automated scripts (shell). With the continuous update of versions and the addition of new language features, it is increasingly used in the development of independent, large-scale projects.

[0060] The following first describes one of the system architectures on which the embodiments of the present invention are based. The image processing method proposed in the embodiments of the present invention can be applied to the system architecture. Figure 1 , Figure 1 is a schematic diagram of an image processing system architecture provided by an embodiment of the present invention, such as Figure 1 As shown, it includes a terminal and a server; the terminal needs to have the functions of taking pictures and communicating (or networking); the picture-taking function can take pictures of vehicles, pedestrians or target objects in a specific area or scene; and the communication (networking) function can send the taken pictures to the server for further processing by the server. The terminal mentioned in the embodiment of the present invention can be a camera, a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile Internet device or other mobile terminal;

[0061] A terminal can be a device at the outermost edge of a computer network, and can also be used for inputting information (such as image data). It can also be called a system, a user unit, a user station, a mobile station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a mobile terminal, a wireless communication device, a user agent, a user device, a plug-in-installable service device, or a user equipment (UE). For example, a terminal can be a cellular phone, a mobile phone, a cordless phone, a smart watch, a wearable device, a tablet device, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication capabilities, a computing device, a vehicle-mounted communication module, a smart meter, or other processing devices connected to a wireless modem with basic functions such as shooting and communication. Figure 1In the system framework shown in , the terminal can upload images to the server according to a certain upload cycle or in real time. The embodiment of the present invention does not limit the number of terminals. Terminal 1 (i.e., port 1), terminal 2 (port 2), ..., terminal N (port N) in the figure are exemplary numbers and do not represent the number of terminals specifically connected to the server or server group.

[0062] A server, or a server group, is a device that provides computing services. Since servers need to respond to service requests and perform processing, they should generally have the ability to undertake and guarantee services. The composition of a server includes a processor, hard disk, memory, system bus, etc., which is similar to the general computer architecture. However, since it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability. In a network environment, servers are divided into file servers, database servers, application servers, WEB servers, etc. according to the different types of services they provide. The server shown in the figure performs algorithmic recognition on the image data according to the target algorithm; specifically, the image data of each checkpoint is collected, and then according to the checkpoint name corresponding to the image and the image size (determined by the image size and image resolution), the images of different checkpoints are saved in the corresponding checkpoint folders according to the image size through a packaged method; according to the different arrangements of the images in each checkpoint image, the images are automatically cut (it can be understood that a camera can take multiple images and the image data stream obtained includes multiple images), and one of them is selected as the labeled image for extracting the labeling information; by labeling the corresponding violation standards in the image, such as the position of the traffic light, the position of the stop line, the position of the solid white line, etc., the relevant labeling information is automatically generated, and according to the relationship between the image resolution, the labeling information is adaptively matched to all the above images through a packaged adaptive algorithm, thereby completing the batch labeling operation.

[0063] Next, read the labeled files according to different folders (corresponding to the collection of pictures taken by multiple cameras), use the script to batch test all the pictures in the folder, and count the required results.

[0064] Finally, after the test is completed, the test results are automatically matched with the expected results, and the parameters required by the following formula are counted to obtain the degree of match between the target algorithm and the application scenario.

[0065] Understandably, Figure 1 The content shown is only an exemplary implementation in the embodiment of the present invention. The system architecture in the embodiment of the present invention may include but is not limited to the above system architecture.

[0066] To facilitate understanding of the embodiments of the present invention, the following exemplary scenarios to which the image processing method in the present application is applied may include a scenario in which a vehicle passes a traffic light in violation of regulations:

[0067] See also Figure 2 , Figure 2 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present invention, the application scenario includes a terminal ( Figure 2 In this example, the terminal is a camera) and a server, and the terminal and the server can be connected through a wireless method such as a network. Figure 2 As shown, when the traffic light in front of the zebra crossing shows that pedestrians can pass, pedestrians cross the zebra crossing to the other side within a preset time period. At this time, vehicles are prohibited from passing the signal light and must stop at a certain distance from the zebra crossing. Assuming that the vehicle runs a red light at this time, that is, the vehicle continues to enter the zebra crossing when the signal light prohibits the vehicle from continuing to move forward; then the camera set on the roadside or the capture device installed on the crossbar of the traffic light will record one or more pictures of the violation. Optionally, when the vehicle leaves the current violation scene, the camera can take one or more pictures of the scene after the violation ends. For example, the pictures taken after the illegal driving behavior occurs can be used to record the current status of the violation scene; the scene may include damage caused by the vehicle to the scene or pedestrians, which is convenient for subsequent accident level assessment. Optionally, when the vehicle appears in the shooting range of the camera, the camera automatically takes one or more pictures. Further optionally, the camera shoots each angle or scene within the shooting range according to a certain shooting cycle.

[0068] Understandably, Figure 2 The application scenarios in the embodiments of the present invention are only several exemplary implementations. The application scenarios in the embodiments of the present invention include but are not limited to the above application scenarios.

[0069] In the following, the technical problems raised in the embodiments of the present invention are specifically analyzed and solved in combination with the above system architecture and the embodiments of the image processing method provided in the embodiments of the present invention.

[0070] See also Figure 3 , Figure 3 This is a flowchart of an image processing method provided by an embodiment of the present invention. The image processing method can be applied to an image processing system (including the above architecture). The image processing method of the embodiment of the present invention can be specifically applied to the scene of traffic violation identification. Figure 3 , taking the server as the execution subject as an example, the method may include the following steps S301-S305; wherein, the optional steps may include step S304 and step S305.

[0071] Step S301: determine a first image set.

[0072] Specifically, after receiving the image data sent by multiple cameras (cameras), the server first classifies the image data to determine the image data captured within a preset time period in a certain scene. The first image set includes multiple first images, and the multiple first images are images captured for the target vehicle in the same scene and the same time period. For example, the image data is first analyzed, and the image is saved to the corresponding camera folder according to the image identifier (for example, the camera number corresponding to the image or the name of the camera that captured a certain scene, and the embodiment of the present invention is described by taking the camera as an example). Then, the pictures in each camera folder are sorted; for example, the pictures in the folder are classified according to the resolution of the picture and the preset classification rules. Pictures with different resolutions have different positions of the corresponding annotated standard information. For example, the resolution of pictures taken during the day and pictures taken at night are different, so in the pictures taken by the same camera, for example, the position of the traffic light may have a certain degree of deviation. In order to facilitate the judgment of the "violation recognition algorithm", the resolution of the picture is preprocessed in advance. For example, the same camera (i.e., multiple images captured by the camera) can be stored in two types of folders, such as images captured during the day and images captured at night.

[0073] Optionally, the two steps of saving the pictures to the corresponding folders and organizing the pictures in the folders can be completed simultaneously. It is understandable that the checkpoint can refer to the camera at the road traffic security checkpoint; for example, at specific places on the road such as toll booths, traffic or security checkpoints, etc., all motor vehicles passing through the point are photographed and recorded. Each camera will have its own camera number; and the type of pictures captured by each camera is fixed, such as the format of the picture, image resolution, picture pixels, etc. The content of the picture type can also include the shooting scene of the camera. For example, if camera A is installed at a highway intersection, then the pictures taken by A will only be relevant pictures of vehicles entering and leaving the highway intersection; for another example, if camera B is installed at a crossroads, then the pictures taken by B are the driving conditions of motor vehicles, non-motor vehicles and pedestrians in this section of the road in different time periods or the capture of illegal driving.

[0074] Optionally, the picture taken by each camera carries the number information of the camera, which is used to indicate which camera took the picture.

[0075] Optionally, the first image in the first image set can be an image with the same image size and image resolution; in the same scene, it can be taken from different angles by different cameras (keeping the image size and image resolution consistent, so that the subsequent server can use this feature as a screening criterion to filter a series of pictures taken in the scene).

[0076] Each of the plurality of first images includes an identifier indicating the first image; and determining the first image set includes: determining the first image set according to the identifier of the first image. Specifically, the identifier of the first image may include a camera number, a shooting scene name, and a name of a violation corresponding to the camera.

[0077] In a possible implementation, a plurality of third pictures are obtained, wherein each of the plurality of third pictures has the same or different image size and image resolution; and the plurality of first pictures having the same image size and image resolution are determined from the plurality of third pictures. The third pictures may include images captured by all cameras. Before determining the first image set, images captured by each camera are obtained.

[0078] Optionally, the pictures are classified according to the area captured by different shooting machines (checkpoints, cameras); or multiple pictures are divided into N categories according to the picture area and picture resolution, wherein each of the N categories corresponds to a preset picture area and picture resolution; each category includes multiple groups of pictures; each group of pictures includes multiple pictures; and each group of pictures corresponds to one illegal act.

[0079] Step S302: determining one or more second image sets from the first image set.

[0080] Specifically, the server selects pictures related to vehicles suspected of driving in violation of the regulations from the first image set, and uses all pictures involved in each suspected violation as the second image set, thereby determining one or more second image sets. That is, the first image set contains images of one or more pending driving violations of one or more target vehicles. Optionally, the camera device of a checkpoint only preliminarily identifies and photographs one type of pending driving violation.

[0081] Optionally, one or more second image sets are determined from a subset of the first image set according to a picture shooting rule. For example, when it is initially identified that vehicle A has committed an illegal behavior (such as running a red light), a picture is taken. The picture is taken when vehicle A enters the field of view and the red light is on; the picture is taken when vehicle A runs through the red light and continues to move forward. The embodiment of the present invention does not limit the number of images taken. In a possible implementation, the shooting rule of the second picture is to take one or more of the second pictures before the illegal driving process occurs; take one or more of the second pictures when the illegal driving process occurs; and take one or more of the second pictures after the illegal driving process occurs.

[0082] For example, the image data stream captured by each checkpoint within a period of time is obtained; the checkpoint will make corresponding marks for the captured image data, and in particular, for illegal behaviors, the checkpoint will distinguish the obtained image data after shooting; for example, in the case where no illegal behavior is detected, the image data captured is marked as A; in the case of shooting illegal behaviors, the obtained image data is marked as B; when shooting after the illegal behavior is over, the obtained image data is marked as C. Optionally, there is at least one image marked as A, B, or C.

[0083] For another example, when there are three types of pictures in total, the pictures at the mount are arranged in one row and three columns; when there are two pictures in total, the pictures at the mount are arranged in two rows and three columns.

[0084] It is understandable that the embodiment of the present invention does not limit the number of identifications, such as ABC, etc. The entire illegal behavior (including before the violation, during the violation, and after the violation) can also be subdivided. In the case of subdivision, the arrangement of the pictures obtained by the checkpoint can be M rows and N columns, that is, the entire process of the illegal behavior is divided into N stages, and each stage has M pictures (M and N are both integers greater than 0). According to the picture data identification corresponding to the picture arrangement of each checkpoint (such as a row of three columns of pictures, a total of three pictures), the picture data stream is cut to obtain multiple groups of pictures (a group of pictures is a set of pictures of one stage); wherein each group of pictures contains at least one picture related to the vehicle. Cutting of pictures mainly depends on the arrangement of the pictures taken by the corresponding checkpoint, which will not be repeated.

[0085] Optionally, when a group of pictures (i.e., a set of images) contains one picture, the group of pictures (i.e., one picture taken at a certain stage) is identified and cut out from the data stream. Optionally, when a group of pictures contains multiple pictures, the group of pictures is cut out first, and then multiple pictures are cut out from the group of pictures. Further optionally, the pictures are classified according to resolution, and after the picture data of the same resolution is cut, a number of pictures of the resolution are sequentially filled into the picture rows and columns of the checkpoint; for example, at an intersection, the checkpoint is a camera at the intersection, which is dedicated to capturing vehicles running red lights. In order to be able to identify the behavior of the vehicle as running a red light, the picture arrangement pattern of the input algorithm is a group of pictures with one row and three columns (for example, a collection of related pictures of illegal behaviors taken during the day with a light intensity of 1000lx, etc.). According to the arrangement of one row and three columns, three pictures with the same resolution (i.e., a certain resolution) are sequentially filled in one row and three columns. For example, picture A is a picture of a vehicle within the stop line, picture B is a picture of a vehicle covering the stop line, and picture C is a picture of a vehicle exceeding the stop line. Subsequent algorithms make judgments based on these three pictures. It is understandable that it is impossible to accurately judge that a vehicle has run a red light based only on pictures A and B.

[0086] Step S303: Mark the reference position of the traffic signs in at least one of the one or more second image sets, and identify the position changes of the target vehicle in the one or more second image sets respectively, so as to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal.

[0087] Specifically, the server marks the traffic signs in the second image contained in each set of the second image set, for example, marking the signal light, solid white line, turn sign, etc. Optionally, in different violation judgment scenarios, different reference positions of traffic signs are set for different violations. For example, in the recognition scenario of running a red light, traffic lights, zebra crossings, etc. are important traffic signs that must be marked.

[0088] In a possible implementation, the marking of the reference position of the traffic sign in at least one of the one or more second image sets, and the identification of the position change of the target vehicle in the one or more second image sets, to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal, includes: marking the reference position of the traffic sign in the target second image set in the one or more second image sets; marking the reference position of the traffic sign in the second image set other than the target second image set in the one or more second image sets according to the reference position of the traffic sign in the target second image set and the position marking algorithm. For example, for the first picture of a checkpoint (i.e., a picture selected to be manually marked), the "illegal standard information" is manually marked, and the "illegal marking information" in the picture is marked, such as the position of the traffic light, the position of the stop line, the position of the solid white line, etc. The aforementioned positions are one of the reference positions for the algorithm to determine whether the vehicle is illegal.

[0089] For example, the server manually annotates the "violation standard information" for the first picture of a checkpoint (i.e., a picture selected to be manually annotated), and annotates the "violation annotation information" in the picture, such as the position of the traffic light, the position of the stop line, the position of the solid white line, etc. The aforementioned positions are one of the reference positions for the algorithm to determine whether the vehicle is illegal. For another example, the traffic recognition algorithm determines whether the vehicle behavior in the input picture group is illegal based on these annotated positions, the traffic rules of the scene, and the preset judgment rules. Furthermore, from the three pictures in the aforementioned row and three columns, select one picture as the annotated picture, and after completing the annotation of the picture, extract the annotation information in the picture. It can be understood that the selected annotated pictures can cover the basic feature values ​​of this type of pictures (such as the group of pictures of running a red light).

[0090] After annotating one image in a group of images, you can annotate other images in the same group. Specifically, for images of a certain resolution of a certain camera, the annotation information on the annotated image is matched to all images of the same resolution through an adaptive algorithm, thereby completing the batch annotation operation. Optionally, by passing the image arrangement format of the adaptive algorithm, a single annotated image containing annotation information and the corresponding image resolution parameters, annotation information can be added to other images of the same resolution of the same camera in batches.

[0091] After the feature extraction of a group of images of a camera is completed, the other groups of images of the camera can be transferred to the same group and the same operations can be performed, such as classification, cutting, labeling and other image preprocessing, so as to complete the batch labeling operation. Among them, the labeling information and image resolution of the labeled images are the parameters of the adaptive algorithm.

[0092] In one possible implementation, the respectively identifying the position changes of the target vehicle in the one or more second image sets to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal includes: respectively identifying the position changes of the target vehicle in the one or more second image sets through a traffic recognition algorithm to obtain the judgment results of the pending illegal driving corresponding to the one or more second image sets; the judgment results include illegal driving and non-illegal driving.

[0093] For example, in different folders, read the labeled files (i.e., standard pictures) and put them into the HTTP request. Then, send the batch-generated annotation information and the tested pictures to the algorithm under test by sending the request. Use a python script to automate the testing of all pictures in all folders and count the results. Specifically, use a python script (containing the algorithm to be evaluated for identifying illegal behavior) to batch test all pictures in a folder. During the test, it is generally necessary to monitor the interface of the target algorithm, analyze the incoming pictures, feedback results, and expected results, and determine the accuracy of the algorithm. It is understandable that the picture recognition algorithm basically provides an interface for picture input to the service user, and returns the corresponding results through the interface, so it is necessary to monitor the incoming pictures and corresponding results of the interface.

[0094] Optionally, the images of different mounts can be saved in the corresponding mount folders according to the mount name and image size through a packaged method; the images can be automatically cut according to the different arrangement modes of the images in each mount image (a camera can take multiple images), and one image is selected as the annotation image for extracting annotation information. The extracted annotation information can be used to annotate images of the same type.

[0095] Step S304: comparing the determination results of pending illegal driving corresponding to the one or more second image sets with the reference results corresponding to the one or more second image sets.

[0096] Specifically, the server compares the results given by the traffic recognition algorithm for the pending illegal driving corresponding to each set in the second image set with the actual judgment results. The actual judgment results may be conclusions drawn based on manual judgment or a certain recognition algorithm with the highest accuracy according to the same method or calculation method. For example, the test results (i.e., the predicted results) are matched with the expected results (i.e., the corresponding actual situations in the table below), wherein the expected results are to manually analyze and determine the data (i.e., the pictures) passed into the algorithm interface to determine whether the behavior involved in the pictures is illegal. The expected results can be obtained directly from a third-party database (such as a road traffic monitoring system, etc.).

[0097] Step S305: Evaluate the accuracy of the traffic recognition algorithm.

[0098] Specifically, the server compares the traffic recognition algorithms through one or more algorithm evaluation indicators. The image is processed according to the above steps, but a different test algorithm is used for each recognition; each algorithm that completes the recognition test will receive a result report to evaluate the matching degree of the algorithm with the current recognition scene.

[0099] In a possible implementation, the evaluating the accuracy of the traffic recognition algorithm includes: evaluating the accuracy of the traffic recognition algorithm by precision, recall, and balanced F score. For example, the parameters required by the following formula are counted to obtain the accuracy of the traffic recognition algorithm. Please refer to Table 1, which lists a comparison table of the test results and the actual results of the traffic recognition algorithm;

[0100] Table 1

[0101]

[0102] Then, the precision is the ratio of samples that are actually positive among the samples predicted as positive by the algorithm:

[0103]

[0104] Among them, P represents the probability of precision, TP is the number of test results that are all true positives, and FP is the number of test results (i.e., the results given by the traffic recognition algorithm for the image) that are false positives (i.e., the actual situation is not a violation, but the algorithm to be tested evaluates that it is a violation). Positive examples are judged to be violations, while negative examples are judged to be non-violations.

[0105] Recall: The proportion of samples predicted as positive by the algorithm on all positive samples is:

[0106]

[0107] Among them, FN is the case where the test algorithm determines that the result is a violation and the actual situation is not a violation. TN is the case where the test algorithm determines that the result is a violation and the actual situation is not a violation.

[0108] The balanced F-score, also known as F1-Score, is defined as the harmonic mean of Precision and Recall.

[0109] as follows:

[0110]

[0111] Use the packaged distribution script to mark the test results on the pictures, and classify and manage the pictures corresponding to different test results; for example, pictures of illegal behaviors are placed in one folder, and pictures of non-illegal behaviors are placed in another folder. Optionally, the pictures of illegal behaviors can also be subdivided into the types of illegal behaviors, and the pictures in the same folder can be stored in folders according to the types of illegal behaviors for easy subsequent viewing.

[0112] Optionally, the relevant records are converted into a test report at the same time, recording the algorithm ratio and the reason for failure.

[0113] Optionally, the test results may include precision, recall, and balanced F-score, as well as other results such as the average, maximum, and minimum time it takes for the algorithm to process a single image.

[0114] The implementation of the embodiment of the present invention improves the efficiency of algorithm evaluation mainly by preprocessing the image data to be tested and analyzing the results of the algorithm. According to the image arrangement of the card slot, the received image data is cut into images with a fixed arrangement. First, a certain image is marked, and then the remaining images with the same resolution are batch-marked based on the marked image. (Similarly, the same operation is performed on images of other resolutions). The marked images are input into the algorithm to be tested to obtain the results; and the matching degree of the algorithm is evaluated based on indicators such as recall rate and precision rate.

[0115] The method of the embodiment of the present invention is described in detail above, and the related device of the embodiment of the present invention is provided below.

[0116] See also Figure 4 , Figure 4 4 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. The image processing device 40 may include a determination unit 401, a screening unit 402, a labeling unit 403 and an evaluation unit 404. Among them, the optional units include the evaluation unit 404.

[0117] A determining unit 401 is used to determine a first image set, where the first image set includes a plurality of first images, and the plurality of first images are images taken of a target vehicle in a same scene and a same time period;

[0118] A screening unit 402 is used to determine one or more second image sets from the first image set, each of the one or more second image sets includes a plurality of second images, and each of the second image sets records a pending illegal driving process of the target vehicle;

[0119] The labeling unit 403 is used to label the reference position of the traffic signs in at least one of the one or more second image sets, and to respectively identify the position changes of the target vehicles in the one or more second image sets, so as to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal.

[0120] In a possible implementation manner, each of the multiple first images includes an identifier indicating the first image; and the determination unit 401 is specifically configured to determine the first image set according to the identifier of the first image.

[0121] In a possible implementation manner, the image size and image resolution of each of the multiple first images are the same.

[0122] In a possible implementation, the labeling unit 403 is specifically configured to:

[0123] Mark the reference positions of traffic signs in a target second image set in the one or more second image sets; and mark the reference positions of traffic signs in second image sets other than the target second image set in the one or more second image sets based on the reference positions of traffic signs in the target second image set and a position marking algorithm.

[0124] In a possible implementation, the labeling unit 403 is specifically used to: identify the position changes of the target vehicle in the one or more second image sets respectively through a traffic recognition algorithm, and obtain the judgment results of the pending illegal driving corresponding to the one or more second image sets respectively; the judgment results include illegal driving and non-illegal driving.

[0125] In a possible implementation, the apparatus further includes an evaluation unit 404, configured to:

[0126] Compare the determination results of the pending illegal driving corresponding to the one or more second image sets with the reference results corresponding to the one or more second image sets; and evaluate the accuracy of the traffic recognition algorithm.

[0127] In a possible implementation, the evaluation unit 404 is specifically configured to:

[0128] The accuracy of the traffic recognition algorithm is evaluated by precision, recall and balanced F-score.

[0129] It should be noted that the functions of the various functional units of the image processing device 40 described in the device embodiment of the present invention can be found in the above Figure 2 The relevant description of the image processing method in the method embodiment is not repeated here.

[0130] The embodiment of the present invention provides an image processing device 50, see Figure 5 , Figure 5 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention, such as Figure 5 As shown, the image processing device 60 can Figure 5 The image processing device 50 may include at least one storage component 501, at least one processing component 502, and at least one communication component 503. In addition, the device may also include common components such as an antenna and a power supply, which will not be described in detail here.

[0131] The storage component 501 may include one or more storage units, each of which may include one or more memories. The storage component may be used to store programs and various data, and may automatically access programs or data at high speed during the operation of the general-purpose device 50. A physical device having two stable states may be used to store information, and the two stable states are represented as "0" and "1" respectively. The aforementioned storage component 501 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (which may include compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0132] The processing component 502 may also be referred to as a processor, a processing unit, a processing board, a processing module, a processing device, etc. The processing component may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP, or may be a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program.

[0133] The communication component 503, which may also be called a transceiver, or a transceiver, etc., may be used to communicate with other devices or communication networks, and may include units for wireless, wired or other communication methods.

[0134] When the image processing device 50 is Figure 1 or Figure 2 When the server is used, the processing component 502 is used to call the data of the storage component 501 to perform the following operations: determine a first image set, the first image set includes multiple first images, and the multiple first images are images taken of the target vehicle in the same scene and the same time period; determine one or more second image sets from the first image set, each of the one or more second image sets includes multiple second images, and each second image set records a pending illegal driving process of the target vehicle; mark the reference position of the traffic sign in at least one of the one or more second image sets, and respectively identify the position change of the target vehicle in the one or more second image sets to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal.

[0135] An embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program includes part or all of the steps of any one of the steps recorded in the above method embodiments.

[0136] An embodiment of the present invention further provides a computer program, which includes instructions. When the computer program is executed by a computer, the computer can execute part or all of the steps of any image processing method.

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0140] In the embodiments of the present invention, the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0141] In addition, the functional components in the embodiments of the present invention may be integrated into one component, or each component may exist physically separately, or two or more components may be integrated into one component. The above integrated components may be implemented in the form of hardware or in the form of software functional units.

[0142] If the integrated components are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0143] The above is only a specific implementation of the embodiment of the present invention, but the protection scope of the embodiment of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the embodiment of the present invention, and these modifications or replacements should be included in the protection scope of the embodiment of the present invention. Therefore, the protection scope of the embodiment of the present invention shall be based on the protection scope of the claims.

[0144] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the present invention. Although the present invention is described in conjunction with various embodiments, in the process of the embodiment of the present invention claimed by the embodiment, those skilled in the art can understand and implement other changes of the disclosed embodiment.

Claims

1. An image processing method, characterized in that: Applied to the evaluation of traffic recognition algorithms, including: Determine a first image set, where the first image set includes a plurality of first images, where the plurality of first images are images taken of a target vehicle in a same scene and within a same time period; Determine one or more second image sets from the first image set, each of the one or more second image sets includes a plurality of second images, and each of the second image sets records a pending illegal driving process of the target vehicle; The reference positions of traffic signs in at least one of the one or more second image sets are marked, and the position changes of the target vehicles in the one or more second image sets are respectively identified to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal. The marking of the reference positions of traffic signs in at least one of the one or more second image sets includes: marking the reference positions of traffic signs in the target second image set in the one or more second image sets; and marking the reference positions of traffic signs in the second image sets other than the target second image set in the one or more second image sets according to the reference positions of traffic signs in the target second image set and a position marking algorithm. The reference positions include traffic light positions, stop line positions, and solid white line positions.

2. The method according to claim 1, characterized in that Each of the plurality of first images includes an identifier indicating the first image; and determining the first image set includes: The first image set is determined according to the identifier of the first image.

3. The method according to claim 1, characterized in that The image size and image resolution of each of the multiple first images are the same.

4. The method according to claim 1, characterized in that: The step of respectively identifying the position change of the target vehicle in the one or more second image sets to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal includes: The position changes of the target vehicle in the one or more second image sets are respectively identified by a traffic recognition algorithm to obtain the judgment results of pending illegal driving corresponding to the one or more second image sets; the judgment results include illegal driving and non-illegal driving.

5. The method according to claim 4, characterized in that The method further comprises: Comparing the determination results of the pending illegal driving corresponding to the one or more second image sets respectively with the reference results corresponding to the one or more second image sets respectively; Evaluate the accuracy of the traffic identification algorithm.

6. The method according to claim 5, characterized in that The evaluating the accuracy of the traffic identification algorithm comprises: The accuracy of the traffic recognition algorithm is evaluated by precision, recall and balanced F-score.

7. An image processing device, the device being used to execute the method according to any one of claims 1 to 6, characterized in that: include: A determination unit, configured to determine a first image set, wherein the first image set includes a plurality of first images, and the plurality of first images are images taken of a target vehicle in a same scene and within a same time period; A screening unit, configured to determine one or more second image sets from the first image set, each of the one or more second image sets comprising a plurality of second images, and each of the second image sets recording a pending illegal driving process of the target vehicle; A labeling unit is used to label the reference position of the traffic sign in at least one of the one or more second image sets, and to identify the position change of the target vehicle in the one or more second image sets, so as to determine whether the pending illegal driving corresponding to the one or more second image sets is illegal.

8. An image processing device, characterized in that: The method comprises a storage component, a communication component and a processing component, wherein the storage component, the communication component and the processing component are interconnected, wherein the storage component is used to store data processing codes, and the communication component is used to exchange information with external devices; the processing component is configured to call program codes to execute the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 6.