Road abnormal object inspection method, device, equipment and medium

Abstract

The invention relates to the technical field of intelligent traffic, and discloses a road abnormal object inspection method, device, equipment and medium, and the method comprises the steps: obtaining continuous multi-frame to-be-detected image data, and determining a region of interest and a background model based on the multi-frame to-be-detected image data; based on the current frame image data and the background model, determining a background album, a foreground image, a connected region, a shadow region and an exclusion region so as to determine a suspected abnormal object in the region of interest; inputting the suspected abnormal object into a pre-trained classification model, and determining information of the suspected abnormal object; and based on the next frame of image data and the suspected abnormal object information, determining the road abnormal object by using the image similarity. According to the scheme, the suspected abnormal object is determined by using the background model and the current frame image data, the classification of the suspected abnormal object is determined by using the classification model, and the suspected abnormal object is judged on the basis of the next frame image data, so that accurate identification of the road abnormal object is ensured.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method, device, equipment and medium for detecting abnormal objects on a road. Background Art

[0002] Autonomous driving systems need to perceive the road environment in real time, including vehicles, pedestrians, road signs, markings, and traffic lights on the road. Abnormal objects on the road may affect the driving safety and route planning of the vehicle, and may even cause the vehicle to stagnate or collide. Therefore, timely detection of abnormal objects on the road is crucial to ensure the safe driving of autonomous vehicles. Abnormal objects refer to various unusual items that appear on the road, such as spilled objects, construction areas, and warning roadblocks.

[0003] In the related art, the background modeling method is usually used to detect these target objects, but the detection of dynamic objects cannot accurately determine their types. For example, what may be detected is water on the road, or it may be a cone or roadblock, etc., and for new abnormal objects that have not been collected, it is difficult for the existing system to accurately detect them, and it is easy to miss detection. Therefore, there is a corresponding demand for accurate identification of road abnormalities in the related art. Summary of the invention

[0004] In view of this, the present invention provides a method, device, equipment, and medium for detecting road abnormalities to solve the technical problem in the related art that road abnormalities cannot be accurately identified.

[0005] In a first aspect, the present invention provides a method for inspecting road anomalies, the method comprising: acquiring a continuous plurality of frames of image data to be tested, and determining an area of ​​interest and a background model based on the plurality of frames of image data to be tested; determining a background album, a foreground image, a connected area, a shadow area and an exclusion area based on current frame image data and a background model in the plurality of frames of image data to be tested; determining suspected anomalies in the area of ​​interest based on the background album, the foreground image, the connected area, the shadow area and the exclusion area; inputting the suspected anomalies into a pre-trained road classification model to determine information about the suspected anomalies; and determining the road anomalies using image similarity based on the next frame image data and the suspected anomaly information in the plurality of frames of image data to be tested.

[0006] In combination with the first aspect, in a possible implementation of the first aspect, the road classification model includes: a pre-trained classification model, a residual network and a fully connected layer, and the suspected abnormal object is input into the pre-trained road classification model to determine the suspected abnormal object information, including: inputting the suspected abnormal object into the classification model, and using the residual network and the fully connected layer to determine the category of the suspected abnormal object; in response to the category of the suspected abnormal object belonging to the abnormal object, recording the coordinates and corresponding image of the suspected abnormal object, and using the coordinates and corresponding image as the suspected abnormal object information.

[0007] In combination with the first aspect, in a possible implementation of the first aspect, based on the next frame of image data in the multiple frames of image data to be tested and the information suspected abnormality, the road abnormality is determined using image similarity, including: calculating the similarity of coordinates based on the next frame of image data in the multiple frames of image data to be tested and the corresponding image; in response to the similarity calculation result exceeding a preset threshold, determining that the suspected abnormality is a road abnormality.

[0008] In combination with the first aspect, in a possible implementation of the first aspect, suspected abnormalities in the region of interest are determined based on the background album, foreground image, connected area, shadow area and exclusion area, including: determining initial suspected abnormalities in the region of interest based on the foreground image, connected area, shadow area and exclusion area; determining suspected abnormalities based on the initial suspected abnormalities and the background album using image similarity.

[0009] In combination with the first aspect, in a possible implementation manner of the first aspect, the method further includes: updating the background album based on current frame image data among the multiple frames of image data to be tested.

[0010] In combination with the first aspect, in a possible implementation manner of the first aspect, the method also includes: inputting multiple frames of image data to be tested into a pre-trained target detection model to determine a first pixel area value and a first pixel size of a road anomaly; inputting multiple frames of known image data into the pre-trained target detection model to determine a second pixel area value and a second pixel size of a known object in the known image; training a learning model using the known image data, the second pixel area, and the second pixel size; and inputting the first pixel area and the first pixel size into the trained learning model to determine an actual area and an actual size of the road anomaly.

[0011] In combination with the first aspect, in a possible implementation method of the first aspect, based on the current frame image data in multiple frames of image data to be tested and the background model, the background album, foreground image, connected area, shadow area and exclusion area are determined, including: determining the target image coordinates of the traffic participant based on the current frame image data in the multiple frames of image data to be tested; using several frames of the multiple frames of image data to be tested as the background album; determining the foreground image based on the current frame image data and the background model; determining the connected area based on the target image coordinates and the foreground image; determining the shadow area by using color conversion based on the background model; and determining the exclusion area based on the area of ​​interest and the current frame image data.

[0012] In a second aspect, the present invention provides a device for inspecting road abnormalities, the device comprising: a multi-frame image module for obtaining continuous multi-frame image data to be tested, and determining an area of ​​interest and a background model based on the multi-frame image data to be tested; an area determination module for determining a background album, a foreground image, a connected area, a shadow area and an exclusion area based on current frame image data and a background model in the multi-frame image data to be tested; a suspected abnormality determination module for determining suspected abnormalities in an area of ​​interest based on the background album, the foreground image, the connected area, the shadow area and the exclusion area; an abnormality information determination module for inputting suspected abnormalities into a pre-trained classification model to determine suspected abnormality information; and a road abnormality determination module for determining road abnormalities using image similarity based on the next frame image data and the suspected abnormality information in the multi-frame image data to be tested.

[0013] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for inspecting road abnormalities according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for inspecting road abnormalities according to the first aspect or any corresponding embodiment thereof.

[0015] The technical solution of the present invention has the following advantages: The present invention provides a method, device, equipment and medium for detecting road anomalies. The method obtains a continuous plurality of frames of image data to be tested, and determines an area of ​​interest and a background model based on the plurality of frames of image data to be tested; determines a background album, a foreground image, a connected area, a shadow area and an excluded area based on the current frame image data and the background model in the plurality of frames of image data to be tested; determines suspected anomalies in the area of ​​interest based on the background album, the foreground image, the connected area, the shadow area and the excluded area; inputs the suspected anomalies into a pre-trained classification model to determine the suspected anomaly information; and determines the road anomaly using image similarity based on the next frame image data and the suspected anomaly information in the plurality of frames of image data to be tested. In this process, the area in the current frame image data is divided by utilizing background modeling, and on this basis, possible suspected abnormal objects are determined, and then the suspected obstacles are classified through the pre-trained classification model, and the similarity is judged again with the corresponding position of the suspected abnormal object in the next frame image, so as to avoid missed detection and false detection problems and ensure the accuracy of detection. Moreover, since this method does not need to rely on the data set, it can still accurately identify new abnormal objects that have not been collected, further ensuring the accuracy of identifying road abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a schematic flow chart of a method for detecting abnormal objects on a road provided according to an embodiment of the present invention; Figure 2 is a structural schematic diagram of a classifier provided according to an embodiment of the present invention; Figure 3 is a structural block diagram of a device for detecting abnormal objects on a road provided according to an embodiment of the present invention; Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] According to an embodiment of the present invention, a method embodiment of road abnormality inspection is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0020] This embodiment provides a method for detecting abnormal objects on a road. Figure 1 As shown, the method comprises the following steps: S101 , acquiring a plurality of continuous frames of image data to be tested, and determining a region of interest and a background model based on the plurality of frames of image data to be tested.

[0021] Specifically, the multiple frames of image data to be tested refer to image data containing multiple frames captured and collected by a camera arranged on the roadside.

[0022] Specifically, based on multiple frames of image data to be tested, determining the region of interest means classifying each pixel point of each frame of the image data to be tested in the multiple frames of image data to be tested through a semantic segmentation model, and performing statistics based on the classification results, so as to determine the region of interest.

[0023] In an optional implementation, determining a region of interest based on multiple frames of image data to be measured includes: Based on multiple frames of image data to be tested, a pre-trained semantic segmentation model is used to determine the classification of each pixel in each frame of the image to be tested and determine the classification result; based on the classification result, a region of interest is determined.

[0024] Specifically, multiple frames of test data are used, and then a target scene semantic segmentation model trained based on a manually annotated data set is used to perform semantic segmentation prediction, so as to obtain the classification of each pixel. After semantic segmentation prediction is performed on multiple frames of test image data, there are N classification results for the pixel at the same position. The N classification results are counted. If the proportion of the classification results that are the area of ​​interest is greater than a threshold, for example, more than 60% of the classification results of a pixel are the area of ​​interest, then the final classification result of the pixel is considered to be the area of ​​interest, and the area of ​​interest can be obtained.

[0025] Specifically, based on the multi-frame image data to be tested, determining the background model means using an illumination compensation algorithm to compensate the collected multi-frame image data to be tested to reduce the influence of illumination, and on this basis using a denoising algorithm to remove image noise to obtain the processed multi-frame image data to be tested, thereby using a Gaussian mixture model to perform background modeling on the processed multi-frame image data to obtain a background model. The denoising algorithm includes but is not limited to a Gaussian blur algorithm, and the background model includes a background album, which is a background album for randomly storing a portion of the processed multi-frame image data to be tested as a background album.

[0026] S102 , based on the current frame image data and the background model in the multiple frames of image data to be tested, determine the background album, foreground image, connected area, shadow area and exclusion area.

[0027] Specifically, based on the current frame image data and the background model in the multiple frames of image data to be tested, determining the background album, foreground image, connected area, shadow area and exclusion area means using the background model to predict the current frame image to obtain the foreground image; using several frame images in the multiple frames of image data to be tested as the background album; determining the connected area by taking the intersection of the foreground image and the target image coordinates; performing color space transformation on the background model and determining the shadow area by using the chromaticity difference; and using the area outside the area of ​​interest in the current frame image as the exclusion area.

[0028] S103, determining suspected abnormal objects in the region of interest based on the background album, the foreground image, the connected area, the shadow area, and the excluded area.

[0029] Specifically, based on the background album, foreground image, connected area, shadow area and excluded area, determining the suspected abnormal object in the region of interest means removing the connected area, shadow area and excluded area in the foreground image, and then using the mathematical morphology algorithm to remove small spots. The spots in the remaining foreground image are the spots corresponding to the screened suspected abnormal objects. The image similarity algorithm is used to compare the similarity between the object corresponding to the spot and the same position in the background album. When the feedback of the corresponding similarity in the background album is low, the suspected abnormal object is further screened out, that is, the suspected abnormal object in the region of interest is determined.

[0030] S104: Input the suspected abnormal object into a pre-trained road classification model to determine the suspected abnormal object information.

[0031] Specifically, inputting the suspected abnormal object into the pre-trained road classification model and determining the suspected abnormal object information means inputting the suspected abnormal object into the pre-trained road classification model, determining the category of the suspected abnormal object, and when the category of the suspected abnormal object belongs to the abnormal object, using the coordinates and corresponding image of the corresponding suspected abnormal object as the suspected abnormal object information.

[0032] S105 , based on the next frame of image data among the multiple frames of image data to be detected and the suspected abnormal object information, the road abnormal object is determined by using image similarity.

[0033] Specifically, based on the next frame of image data in multiple frames of image data to be tested and the suspected abnormal object, determining the road abnormal object by using image similarity means calculating the similarity between the image at the corresponding position of the suspected abnormal object in the next frame of image and the suspected abnormal object, and when the similarity exceeds a preset threshold, the corresponding suspected abnormal object is considered to be a road abnormal object.

[0034] The present invention provides a method, device, equipment and medium for detecting road anomalies. The method obtains a continuous plurality of frames of image data to be tested, and determines an area of ​​interest and a background model based on the plurality of frames of image data to be tested; determines a background album, a foreground image, a connected area, a shadow area and an excluded area based on the current frame image data and the background model in the plurality of frames of image data to be tested; determines suspected anomalies in the area of ​​interest based on the background album, the foreground image, the connected area, the shadow area and the excluded area; inputs the suspected anomalies into a pre-trained classification model to determine the suspected anomaly information; and determines the road anomaly using image similarity based on the next frame image data and the suspected anomaly information in the plurality of frames of image data to be tested. In this process, the area in the current frame image data is divided by utilizing background modeling, and on this basis, possible suspected abnormal objects are determined, and then the suspected obstacles are classified through the pre-trained classification model, and the similarity is judged again with the corresponding position of the suspected abnormal object in the next frame image, so as to avoid missed detection and false detection problems and ensure the accuracy of detection. Moreover, since this method does not need to rely on the data set, it can still accurately identify new abnormal objects that have not been collected, further ensuring the accuracy of identifying road abnormalities.

[0035] In an optional implementation, the road classification model includes: a pre-trained classification model, a residual network and a fully connected layer, and the suspected abnormal object is input into the pre-trained road classification model to determine the suspected abnormal object information, including: The suspected anomaly is input into the classification model, and the category of the suspected anomaly is determined by using the residual network and the fully connected layer; in response to the category of the suspected anomaly belonging to the abnormal object, the coordinates and the corresponding image of the suspected anomaly are recorded, and the coordinates and the corresponding image are used as the suspected anomaly information.

[0036] Specifically, the suspected anomaly is input into the pre-trained road classification model. Determining the category of the suspected anomaly means expanding the image spot corresponding to the suspected anomaly to a certain pixel value and adjusting the image spot size to the image input size of the road classification model, and then inputting the pre-trained road classification model to obtain the category of the suspected anomaly, that is, judging whether the suspected anomaly is an anomaly, thereby reducing false alarms caused by shadows, puddles, etc. Figure 2 As shown, the road classification model includes: a pre-trained classification model, a residual network and a fully connected layer. Among them, the pre-trained classification model can be a CLIP model, such as a CLIP-Adapter model. The classifier used in this embodiment uses the pre-trained classification model as the backbone network, and connects two residual network structure layers and a fully connected layer after obtaining the image feature layer. Therefore, on the one hand, the classification ability of the pre-trained model for conventional objects can be effectively utilized. On the other hand, the accurate classification effect of the target can be achieved by training the subsequent residual network and fully connected layer, and the generalization ability of the model is maintained. When constructing the data set, road classification is the main focus, and road anomalies and conventional samples with large feature differences from the above two are used as non-target classification as a supplement. When there is ambiguity in the classification of anomalies, the sample ratio and sample characteristics of the second and third categories of targets can be appropriately adjusted.

[0037] Specifically, in response to the suspected abnormal object being classified as an abnormal object, recording the coordinates of the suspected abnormal object and the corresponding image means that when the suspected abnormal object is classified as an abnormal object, the coordinate position of the abnormal object in the image and the image slice of the coordinate position are saved and recorded to form suspected abnormal object information.

[0038] In an optional implementation, based on the next frame of image data in the multiple frames of image data to be detected and the information suspected abnormal object, the road abnormal object is determined by using image similarity, including: Based on the next frame of image data in the multiple frames of image data to be tested and the corresponding image, the similarity of the coordinates is calculated; in response to the similarity calculation result exceeding a preset threshold, the suspected abnormal object is determined to be a road abnormal object.

[0039] Specifically, based on the next frame of image data and the image in multiple frames of image data to be tested, the similarity of the coordinates is calculated, and determining the calculation result means calculating the image similarity of the image at the corresponding coordinate position in the next frame of image and the pattern corresponding to the abnormal object in the current frame of image, and determining the similarity calculation result.

[0040] Specifically, the preset threshold value can be set according to actual working conditions, and this embodiment does not make any specific limitation to this.

[0041] It should be understood that in the related art, in order to further screen suspected road anomaly spots, it is necessary to save the slices, names and coordinates of the anomaly into the database, and determine whether there are anomalies with the same name and coordinates in the database, so that when there are, the number of records is increased by 1, and when the number of records is greater than the preset number, it is considered that the object has been in one place for a sufficient period of time, and on this basis, the similarity of the slices of the object and the same position in the background album is calculated using image similarity, so that the object is considered to be an anomaly. In this way, the obtained spots have the problem of gradually disappearing, because the foreground object will be gradually integrated into the background modeling, and the image processing of the mathematical morphology algorithm will cause the spots of the foreground object to be reduced, resulting in missed detection of anomalies.

[0042] The present embodiment effectively compares the images by introducing a pre-trained road classifier model and comparing the next frame of the multiple frames of image data to be tested, and avoids the problem of the gradual disappearance of the spots existing in the related art, thereby effectively ensuring the accuracy of identifying abnormal objects on the road. That is, through the pre-trained road classifier model, whether the suspected abnormal object is an abnormal object is classified and identified, and because of the introduction of multiple frames of image data to be tested, dynamic objects that accidentally fly into the camera screen, such as flying birds or other objects, can be prevented from being misdetected as abnormal objects on the road. This further illustrates that the introduction of a pre-trained road classifier model and the comparison of multiple frames of image data to be tested can accurately ensure the accuracy of identifying abnormal objects on the road.

[0043] In an optional implementation, based on the background album, the foreground image, the connected area, the shadow area, and the excluded area, determining the suspected abnormal object in the region of interest includes: Based on the foreground image, the connected area, the shadow area and the excluded area, the initial suspected anomaly in the region of interest is determined; based on the initial suspected anomaly and the background album, the suspected anomaly is determined by using image similarity.

[0044] Specifically, based on the foreground image, the connected area, the shadow area and the excluded area, determining the initial suspected anomaly in the area of ​​interest means removing the connected area, the shadow area and the excluded area in the foreground image, and then using a mathematical morphology algorithm to remove small spots. The spots remaining in the foreground image are the spots corresponding to the initial suspected anomalies after screening.

[0045] Specifically, based on the initial suspected abnormal object and the background album, determining the suspected abnormal object by using the image similarity means using the position of the image spot corresponding to the initial suspected abnormal object, the corresponding position in the background album, and calculating the image similarity between the two. When the image similarity exceeds the corresponding similarity threshold of the image data to be tested exceeds the preset number, the corresponding initial suspected abnormal object is considered to be a suspected abnormal object. For example, if the background album has 3 frames of image data to be tested, the position of the initial suspected abnormal object A is a1, the position of the initial suspected abnormal object B is b1, the similarity calculation results between the initial suspected abnormal object A and the position a1 in the background album are d1, d2, and d3 respectively, and the similarity calculation results between the initial suspected abnormal object B and the position b1 in the background album are d4, d5, and d6 respectively, where d1, d2, and d6 are greater than the similarity threshold, and the preset number is 60%, then the initial suspected abnormal object A is a suspected abnormal object, and the initial suspected abnormal object B is not a suspected abnormal object.

[0046] In an optional implementation, the method further includes: updating the background album based on the current frame image data among the multiple frames of image data to be tested.

[0047] Specifically, updating the background model and the background album based on the current frame image data in the multiple frames of image data to be tested refers to updating the background model and the background album using the current frame image data, wherein the updating principle of the background album is that the updating operation is probabilistic.

[0048] In an optional implementation, the background album includes: a first part, a second part, and a third part. The update probability of the first part is lower than that of the second part, and the update probability of the second part is lower than that of the third part.

[0049] Specifically, the length of the background album is divided into three parts, namely the first part, the second part, and the third part, which are recorded as section A, section B, and section C. The background album of section A corresponds to a section with a very low probability of being updated, the background album of section B has a higher probability of being updated than section A, and the background album of section C has the highest probability of being updated, so that the background album is segmented so that the background album has the ability to record the longest possible background history.

[0050] In an optional embodiment, the method further comprises: Input multiple frames of image data to be tested into a pre-trained target detection model to determine a first pixel area value and a first pixel size of a road anomaly; input multiple frames of known image data into the pre-trained target detection model to determine a second pixel area value and a second pixel size of a known object in the known image; train a learning model using the known image data, the second pixel area, and the second pixel size; input the first pixel area and the first pixel size into the trained learning model to determine an actual area and an actual size of the road anomaly.

[0051] Specifically, based on multiple frames of image data to be tested, inputting a pre-trained target detection model, determining the first pixel area value and the first pixel size of the road anomaly means using the target detection model trained based on the manually annotated data set to predict the image using the multiple frames of image data to be tested, and predicting the pixel area value corresponding to the target box composed of each coordinate point of the target, and the corresponding pixel size, that is, the first pixel area value and the first pixel size. Among them, the target detection model refers to a detector dedicated to a certain type of target. If a box is used as a reference object for calibrating the size of objects at each position, the pre-trained target detection model is a trained box detector. Similarly, if a vehicle is used as a reference object for calibrating the size of objects at each position, a vehicle detector is trained. The first pixel area value refers to the area value composed of pixels corresponding to the object in the multiple frames of image data to be tested in the picture. Similarly, the first pixel size refers to the size value composed of pixels corresponding to the object in the multiple frames of image data to be tested in the picture.

[0052] Specifically, using multiple frames of known image data and inputting a pre-trained target detection model to determine the second pixel area value and the second pixel size of a known object in the known image means using the pre-trained target detection model to determine the pixel area and pixel size of the known object at different positions in the image field of view, that is, the second pixel area value and the second pixel size.

[0053] Specifically, using known image data, the second pixel area, and the second pixel size to train the learning model refers to using the actual size and actual area value of the known object in the physical coordinate system in multiple frames of known image data, such as a cubic box with a volume of 1 cubic meter, 0.512 cubic meters, and 0.125 cubic meters, and using a shallow neural network model or a machine learning model to obtain a pixel area model that predicts the actual size based on the image coordinates, that is, a learning model. Among them, the shallow neural network model includes the FCN model, and the machine learning model includes the RF fitting model. Thus, the learning model is used to obtain a linear relationship between the actual size and actual area of ​​the object and the pixel size and pixel area based on the object being placed at different positions in the image field of view.

[0054] Specifically, inputting the first pixel area and the first pixel size into the trained learning model to determine the actual area and the actual size of the road anomaly means that after the learning model has mastered the linear relationship between the actual size and the actual area of ​​the object and the pixel size and the pixel area, the first pixel area and the first pixel size are input into the trained learning model to obtain the actual area and the actual size of the road anomaly. It should be understood that by determining the actual size and the actual area of ​​the road anomaly, a data basis can be provided for the subsequent screening of the target object, which facilitates the further screening of the road anomaly, thereby providing a data basis for the vehicle to provide decision-making information. For example, when it is necessary to screen road obstacles with an actual size greater than 50cm, the actual size of the road anomaly can be determined to determine the length of the road section affected by the road anomaly in the actual road or the width of the road section affected, thereby sending a decision information such as lane change to the vehicle in advance to avoid the vehicle's travel being affected. For another example, it is necessary to screen a single lane area that occupies more than 20cm 2 When there is an obstacle on the road, the actual area of ​​the road anomaly can be determined, and the area affected by the road anomaly in the actual road can be determined, so as to send the decision information to the vehicle in advance that there is water on the road but it will not affect the normal passage of the vehicle or lane change, so as to avoid the vehicle's movement being affected.

[0055] In an optional implementation, based on the current frame image data and the background model in the multiple frames of image data to be tested, the background album, the foreground image, the connected area, the shadow area and the excluded area are determined, including: Based on the current frame image data in the multiple frames of image data to be tested, the target image coordinates of the traffic participants are determined; based on the current frame image data and the background model, the foreground image is determined; based on the target image coordinates and the foreground image, the connected area is determined; based on the background model, the shadow area is determined by color conversion; based on the area of ​​interest and the current frame image data, the exclusion area is determined.

[0056] Specifically, based on the current frame image data in the multiple frames of image data to be tested, the target image coordinates of the traffic participant are determined, and the current frame image data is input into the pre-trained image detection model to determine the type of traffic participant and the corresponding coordinates, that is, the target image coordinates of the traffic participant, which are expressed as the coordinate position corresponding to the target box, wherein the types of traffic participants include: pedestrians, motor vehicles, and non-motor vehicles.

[0057] Specifically, using several frames of the multiple frames of image data to be tested as the background album means randomly storing a portion of the multiple frames of image data to be tested as the background album, that is, randomly storing a portion of the processed multiple frames of image data to be tested as the background album.

[0058] Specifically, based on the current frame image data and the background model, determining the foreground image means predicting the current frame image using the background model to determine the foreground image.

[0059] Specifically, based on the target image coordinates and the foreground image, determining the connected region means taking the union of the foreground image coordinates and the target image coordinates to obtain the connected region of the two results, which is denoted as F1.

[0060] Specifically, based on the background model, using color conversion, determining the shadow area means converting the RGB color space into the LUV color space by color space transformation, and balancing the L value to obtain the background model and the current frame in chromaticity close, with a large brightness difference in illumination and shadow area, that is, the shadow area, denoted as F2.

[0061] Specifically, based on the region of interest and the current frame image, determining the exclusion region means taking the region other than the region of interest in the current frame image as the exclusion region, which is denoted as F3.

[0062] In this embodiment, a device for detecting abnormal objects on a road is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0063] This embodiment provides a device for detecting abnormal objects on a road, such as Figure 3 As shown, including: The multi-frame image module 201 is used to obtain continuous multi-frame image data and determine the region of interest and the background model based on the multi-frame image data. The specific process can be found in the description of step S101 in the above embodiment, which will not be repeated here.

[0064] The region determination module 202 is used to determine the background album, foreground image, connected area, shadow area and exclusion area based on the current frame image data and the background model in the multiple frames of image data to be tested. The specific process can be referred to the relevant description of step S102 in the above embodiment, which will not be repeated here.

[0065] The suspected abnormal object determination module 203 is used for the suspected abnormal object determination module, which is used to determine the suspected abnormal object in the region of interest based on the background album, the foreground image, the connected area, the shadow area and the excluded area. The specific process can be referred to the relevant description of step S103 in the above embodiment, which will not be repeated here.

[0066] The abnormal object information determination module 204 is used to input the suspected abnormal object into the pre-trained classification model to determine the suspected abnormal object information. The specific process can be referred to the relevant description of step S104 in the above embodiment, which will not be repeated here.

[0067] The road anomaly determination module 205 is used to determine the road anomaly based on the next frame of image data in the multiple frames of image data to be detected and the suspected anomaly information by using image similarity. The specific process can be found in the description of step S105 in the above embodiment, which will not be repeated here.

[0068] The device for inspecting road abnormalities in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0069] The embodiment of the present invention also provides a computer device having the above Figure 3 The device for detecting abnormal objects on the road is shown.

[0070] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 301, memory 302, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 301 is taken as an example.

[0071] The processor 301 may be a central processing unit, a network processor or a combination thereof. The processor 301 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.

[0072] The memory 302 stores instructions executable by at least one processor 301, so that the at least one processor 301 executes the method shown in the above embodiment.

[0073] The memory 302 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 302 may optionally include a memory remotely arranged relative to the processor 301, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] The memory 302 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 302 may also include a combination of the above types of memory. The computer device also includes a communication interface 303, which is used for the computer device to communicate with other devices or communication networks.

[0075] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0076] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for detecting abnormal objects on a road, characterized in that: The method comprises: Acquire a plurality of continuous frames of image data to be tested, and determine a region of interest and a background model based on the plurality of frames of image data to be tested; Based on the current frame image data in the multiple frames of image data to be tested and the background model, determine the background album, foreground image, connected area, shadow area and exclusion area; Determine suspected abnormal objects in the region of interest based on the background album, the foreground image, the connected area, the shadow area, and the excluded area; Inputting the suspected abnormal object into a pre-trained road classification model to determine the suspected abnormal object information; Based on the next frame of image data among the multiple frames of image data to be detected and the suspected abnormal object information, the road abnormal object is determined by utilizing image similarity.

2. The method according to claim 1, characterized in that The road classification model includes: a pre-trained classification network, a residual network and a fully connected layer. The suspected abnormal object is input into the pre-trained road classification model to determine the suspected abnormal object information, including: Inputting the suspected anomaly into the classification network, and using the residual network and the fully connected layer to determine the category of the suspected anomaly; In response to the category of the suspected abnormal object belonging to the abnormal object, the coordinates and the corresponding image of the suspected abnormal object are recorded, and the coordinates and the corresponding image are used as the suspected abnormal object information.

3. The method according to claim 2, characterized in that Determining the road abnormality based on the next frame of image data in the plurality of frames of image data to be detected and the information suspected abnormality by using image similarity includes: Calculating the similarity of the coordinates based on the next frame of image data in the plurality of frames of image data to be tested and the corresponding image; In response to the similarity calculation result exceeding a preset threshold, the suspected abnormal object is determined to be a road abnormal object.

4. The method according to claim 1, characterized in that: The determining of suspected abnormal objects in the region of interest based on the background album, the foreground image, the connected area, the shadow area, and the excluded area includes: Determine an initial suspected anomaly in a region of interest based on the foreground image, the connected region, the shadow region, and the excluded region; Based on the initial suspected anomaly and the background album, the suspected anomaly is determined by utilizing image similarity.

5. The method according to claim 4, characterized in that Also includes: The background album is updated based on the current frame image data among the multiple frames of image data to be tested.

6. The method according to claim 1, characterized in that The method further comprises: Inputting multiple frames of image data to be tested into a pre-trained target detection model to determine a first pixel area value and a first pixel size of the road anomaly; Inputting multiple frames of known image data into a pre-trained object detection model to determine a second pixel area value and a second pixel size of a known object in the known image; Training a learning model using the known image data, the second pixel area, and the second pixel size; The first pixel area and the first pixel size are input into the trained learning model to determine the actual area and the actual size of the road anomaly.

7. The method according to claim 1, characterized in that The determining of the background album, the foreground image, the connected area, the shadow area and the excluded area based on the current frame image data in the multiple frames of image data to be tested and the background model comprises: Determining target image coordinates of the traffic participant based on the current frame image data among the multiple frames of image data to be tested; Using several frames of images among the multiple frames of image data to be tested as background albums; Determine a foreground image based on the current frame image data and the background model; Determine a connected area based on the target image coordinates and the foreground image; Based on the background model, using color conversion, determining the shadow area; An exclusion area is determined based on the region of interest and the current frame image data.

8. A device for detecting abnormal objects on a road, characterized in that: The device comprises: A multi-frame image module for testing, used for acquiring continuous multi-frame image data for testing, and determining a region of interest and a background model based on the multi-frame image data for testing; An area determination module, used for determining a background album, a foreground image, a connected area, a shadow area and an excluded area based on the current frame image data in the multiple frames of image data to be tested and the background model; A suspected abnormal object determination module, used to determine the suspected abnormal object in the region of interest based on the background album, the foreground image, the connected area, the shadow area and the excluded area; An abnormal object information determination module, used for inputting the suspected abnormal object into a pre-trained classification model to determine the suspected abnormal object information; The road abnormality determination module is used to determine the road abnormality based on the next frame of image data in the multiple frames of image data to be detected and the suspected abnormality information by using image similarity.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for detecting road abnormalities according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for inspecting road abnormalities according to any one of claims 1 to 7.

Images