Road foreign matter detection method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of deep learning, and particularly discloses a road foreign matter detection method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-detected image containing a target road region, and extracting a long-time background image and a short-time background image corresponding to the to-be-detected image; according to the to-be-detected image, the long-time background image and the short-time background image, a road foreign matter detection result of the to-be-detected image is obtained by using the trained road foreign matter detection model; wherein the road foreign matter detection model comprises a convolutional neural network used for extracting image prior information and a target detection network used for road foreign matter detection. According to the invention, road foreign matter detection is carried out by combining the long and short time backgrounds of the road image and the prior information, the problem of low precision caused by collection of a large number of sample images of road abnormal objects is avoided, and the accuracy of road foreign matter detection is improved while the method adapts to road foreign matter detection in different complex environment scenes.

Description

Background Art

[0002] There are the following problems in road foreign object detection: 1) Due to the uncertainty of the properties of abnormal objects, it takes a lot of manpower and financial resources to collect as many abnormal objects as possible, which is not realistic. 2) The road environment is an outdoor environment, which is complex and changeable. The scene changes with the time of day and is affected by the seasons and weather (cloudy, sunny, foggy, etc.). 3) Traditional detection methods need to be based on a reliable and real background. Compared with indoor environment applications, how to extract a background with strong robustness in this complex and changeable outdoor road scene is very challenging. When the background is not updated or the update rate is slow, the changes in illumination and shadows on the road surface may lead to false detections, and if there is a target in the background, "ghosts" may also be generated. When the background is updated frequently, static abnormal objects may be updated into the background, resulting in missed detections. 4) Abnormal object detection based on deep learning (such as YOLO series, RCNN series, etc.) can only output abnormal objects of fixed categories, and is powerless for abnormal objects outside its category range.

[0003] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method, system, electronic device and storage medium for detecting foreign objects on a road.

[0005] In a first aspect, the present invention provides a method for detecting foreign objects on a road, and the technical solution of the method is as follows:

[0006] Acquire an image to be detected that includes a target road area, and extract a long-term background image and a short-term background image corresponding to the image to be detected;

[0007] According to the image to be detected, the long-term background image and the short-term background image, and using a trained road foreign object detection model, a road foreign object detection result of the image to be detected is obtained; wherein the road foreign object detection model includes: a convolutional neural network for extracting image prior information and a target detection network for road foreign object detection.

[0008] The beneficial effects of a road foreign object detection method of the present invention are as follows:

[0009] The method of the present invention combines the long-term and short-term backgrounds of road images and prior information to detect road foreign objects, thereby avoiding the problem of low accuracy caused by collecting a large number of sample images of abnormal road objects. While adapting to road foreign object detection in different complex environmental scenarios, it improves the accuracy of road foreign object detection.

[0010] The long- and short-term background and prior information corresponding to the road image can improve the accuracy of road foreign object detection.

[0011] Based on the above solution, the road foreign object detection method of the present invention can also be improved as follows.

[0012] In an optional manner, the step of obtaining the long-term background image and the short-term background image corresponding to the image to be detected includes:

[0013] The long-term background image corresponding to the image to be detected is obtained by using a first Gaussian mixture model, and the short-term background image corresponding to the image to be detected is obtained by using a second Gaussian mixture model; wherein both the first Gaussian mixture model and the second Gaussian mixture model are trained using sample images containing the target road area, and the update frequency of the first Gaussian mixture model is less than the update frequency of the second Gaussian mixture model.

[0014] In an optional manner, the convolutional neural network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a first grouped convolutional layer, a second grouped convolutional layer, and a third grouped convolutional layer; the target detection network includes: a first Backbone layer, a second Backbone layer, a third Backbone layer, a first channel connection layer, a second channel connection layer, a third channel connection layer, a first Neck layer, a second Neck layer, a first Head layer, a second Head layer, and a third Head layer;

[0015] The first convolution layer is used to perform convolution processing on the target input image to obtain a first image feature; wherein the target input image is a nine-channel image composed of the image to be detected, the long-term background image and the short-term background image; the first grouped convolution layer is used to perform grouped convolution processing on the first image feature to obtain a second image feature; the second convolution layer is used to perform convolution processing on the second image feature to obtain a third image feature; the third convolution layer is used to perform convolution processing on the first image feature to obtain a fourth image feature; the second grouped convolution layer is used to perform grouped convolution processing on the fourth image feature to obtain a fifth image feature; the fourth convolution layer is used to perform convolution processing on the fifth image feature to obtain a sixth image feature; the fifth convolution layer is used to perform convolution processing on the fourth image feature to obtain a seventh image feature; the third grouped convolution layer is used to perform grouped convolution processing on the seventh image feature to obtain an eighth image feature; the sixth convolution layer is used to perform convolution processing on the eighth image feature to obtain a ninth image feature;

[0016] The first Backbone layer is used to perform feature extraction on the target input image to obtain a tenth image feature; the first channel connection layer is used to perform channel connection on the third image feature and the tenth image feature to obtain an eleventh image feature; the second Backbone layer is used to perform feature extraction on the eleventh image feature to obtain a twelfth image feature; the second channel connection layer is used to perform channel connection on the sixth image feature and the twelfth image feature to obtain a thirteenth image feature; the third Backbone layer is used to perform feature extraction on the thirteenth image feature to obtain a fourteenth image feature; the third channel connection layer is used to perform channel connection on the ninth image feature and the fourteenth image feature to obtain a fifteenth image feature. image features; the first Neck layer is used to: perform feature fusion on the eleventh image feature, the twelfth image feature and the fourteenth image feature to obtain the sixteenth image feature; the first Head layer is used to: detect the sixteenth image feature to obtain a first detection result; the second Neck layer is used to: perform feature fusion on the thirteenth image feature and the fourteenth image feature to obtain the seventeenth image feature; the second Head layer is used to: detect the seventeenth image feature to obtain a second detection result; the third Head layer is used to: detect the fifteenth image feature to obtain a third detection result; wherein the first detection result, the second detection result and the third detection result together constitute the road foreign object detection result.

[0017] In an optional manner, the road foreign object detection result is: a target detection frame containing a foreign object in the image to be detected and a probability of the target detection frame.

[0018] In a second aspect, the present invention provides a road foreign object detection system, the technical solution of which is as follows:

[0019] Including: acquisition module and detection module;

[0020] The acquisition module is used to: acquire an image to be detected that includes a target road area, and extract a long-term background image and a short-term background image corresponding to the image to be detected;

[0021] The detection module is used to obtain a road foreign object detection result of the image to be detected based on the image to be detected, the long-term background image and the short-term background image, and using a trained road foreign object detection model; wherein the road foreign object detection model includes: a convolutional neural network for extracting image prior information and a target detection network for road foreign object detection.

[0022] The beneficial effects of a road foreign object detection system of the present invention are as follows:

[0023] The system of the present invention detects foreign objects on the road by combining the long-term and short-term backgrounds of the road image and prior information, thereby avoiding the problem of low accuracy caused by collecting a large number of sample images of abnormal road objects. While adapting to the detection of foreign objects on the road in different complex environmental scenarios, it improves the accuracy of the detection of foreign objects on the road.

[0024] Based on the above solution, the road foreign object detection system of the present invention can also be improved as follows.

[0025] In an optional manner, the acquisition module is specifically used to:

[0026] The long-term background image corresponding to the image to be detected is obtained by using a first Gaussian mixture model, and the short-term background image corresponding to the image to be detected is obtained by using a second Gaussian mixture model; wherein both the first Gaussian mixture model and the second Gaussian mixture model are trained using sample images containing the target road area, and the update frequency of the first Gaussian mixture model is less than the update frequency of the second Gaussian mixture model.

[0027] In an optional manner, the convolutional neural network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a first grouped convolutional layer, a second grouped convolutional layer, and a third grouped convolutional layer; the target detection network includes: a first Backbone layer, a second Backbone layer, a third Backbone layer, a first channel connection layer, a second channel connection layer, a third channel connection layer, a first Neck layer, a second Neck layer, a first Head layer, a second Head layer, and a third Head layer;

[0028] The first convolution layer is used to perform convolution processing on the target input image to obtain a first image feature; wherein the target input image is a nine-channel image composed of the image to be detected, the long-term background image and the short-term background image; the first grouped convolution layer is used to perform grouped convolution processing on the first image feature to obtain a second image feature; the second convolution layer is used to perform convolution processing on the second image feature to obtain a third image feature; the third convolution layer is used to perform convolution processing on the first image feature to obtain a fourth image feature; the second grouped convolution layer is used to perform grouped convolution processing on the fourth image feature to obtain a fifth image feature; the fourth convolution layer is used to perform convolution processing on the fifth image feature to obtain a sixth image feature; the fifth convolution layer is used to perform convolution processing on the fourth image feature to obtain a seventh image feature; the third grouped convolution layer is used to perform grouped convolution processing on the seventh image feature to obtain an eighth image feature; the sixth convolution layer is used to perform convolution processing on the eighth image feature to obtain a ninth image feature;

[0029] The first Backbone layer is used to perform feature extraction on the target input image to obtain a tenth image feature; the first channel connection layer is used to perform channel connection on the third image feature and the tenth image feature to obtain an eleventh image feature; the second Backbone layer is used to perform feature extraction on the eleventh image feature to obtain a twelfth image feature; the second channel connection layer is used to perform channel connection on the sixth image feature and the twelfth image feature to obtain a thirteenth image feature; the third Backbone layer is used to perform feature extraction on the thirteenth image feature to obtain a fourteenth image feature; the third channel connection layer is used to perform channel connection on the ninth image feature and the fourteenth image feature to obtain a fifteenth image feature. image features; the first Neck layer is used to: perform feature fusion on the eleventh image feature, the twelfth image feature and the fourteenth image feature to obtain the sixteenth image feature; the first Head layer is used to: detect the sixteenth image feature to obtain a first detection result; the second Neck layer is used to: perform feature fusion on the thirteenth image feature and the fourteenth image feature to obtain the seventeenth image feature; the second Head layer is used to: detect the seventeenth image feature to obtain a second detection result; the third Head layer is used to: detect the fifteenth image feature to obtain a third detection result; wherein the first detection result, the second detection result and the third detection result together constitute the road foreign object detection result.

[0030] In an optional manner, the road foreign object detection result is: a target detection frame containing a foreign object in the image to be detected and a probability of the target detection frame.

[0031] In a third aspect, a technical solution of an electronic device of the present invention is as follows:

[0032] The invention comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the road foreign object detection method of the present invention are implemented.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0034] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the road foreign object detection method of the present invention.

[0035] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0037] Figure 1 A schematic diagram of a process flow of an embodiment of a method for detecting foreign objects on a road according to the present invention;

[0038] Figure 2 It is a schematic diagram of the overall process of the road foreign object detection model;

[0039] Figure 3 A schematic structural diagram of an embodiment of a road foreign object detection system of the present invention;

[0040] Figure 4 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0041] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0042] Figure 1The flowchart of an embodiment of a method for detecting foreign objects on a road provided by the present invention is shown. The method for detecting foreign objects on a road can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the method for detecting foreign objects on a road by calling computer-readable instructions stored in a memory through a processor. Figure 1 As shown, the following steps are included:

[0043] S1. Acquire an image to be detected that includes a target road area, and extract a long-term background image and a short-term background image corresponding to the image to be detected.

[0044] Among them, the target road area is: the road area to be detected for foreign objects in this embodiment. Foreign objects refer to objects other than common objects on the road such as vehicles, pedestrians, and traffic signs. The long-term background image refers to a collection of pixels in which the grayscale value of the background part of the image hardly changes or only changes slightly. It usually represents the real external environment and remains stable for a long time. The short-term background image refers to an image in which the background part of the image is captured or generated in a short time, which may contain more dynamic changes and noise. The long-term background image pays more attention to stability and accuracy, while the short-term background image pays more attention to real-time and dynamics.

[0045] S2. Obtain a road foreign object detection result of the image to be detected based on the image to be detected, the long-term background image and the short-term background image, and using a trained road foreign object detection model.

[0046] Among them, the road foreign object detection model includes: a convolutional neural network for extracting image prior information and a target detection network for road foreign object detection.

[0047] In an optional manner, the step of obtaining a long-term background image and a short-term background image corresponding to the image to be detected includes:

[0048] The first Gaussian mixture model is used to obtain a long-term background image corresponding to the image to be detected, and the second Gaussian mixture model is used to obtain a short-term background image corresponding to the image to be detected.

[0049] Wherein, both the first Gaussian mixture model and the second Gaussian mixture model are trained using sample images containing the target road area, and the update frequency of the first Gaussian mixture model is less than the update frequency of the second Gaussian mixture model.

[0050] In an alternative approach, Figure 2 As shown, the convolutional neural network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a first group convolutional layer, a second group convolutional layer and a third group convolutional layer; the target detection network includes: a first Backbone layer, a second Backbone layer, a third Backbone layer, a first channel connection layer, a second channel connection layer, a third channel connection layer, a first Neck layer, a second Neck layer, a first Head layer, a second Head layer and a third Head layer.

[0051] The first convolution layer is used to: perform convolution processing on the target input image to obtain the first image feature; wherein the target input image is: a nine-channel image composed of the image to be detected, the long-time background image and the short-time background image; the first grouped convolution layer is used to: perform grouped convolution processing on the first image feature to obtain the second image feature; the second convolution layer is used to: perform convolution processing on the second image feature to obtain the third image feature; the third convolution layer is used to: perform convolution processing on the first image feature to obtain the fourth image feature; the second grouped convolution layer is used to: perform grouped convolution processing on the fourth image feature to obtain the fifth image feature; the fourth convolution layer is used to: perform convolution processing on the fifth image feature to obtain the sixth image feature; the fifth convolution layer is used to: perform convolution processing on the fourth image feature to obtain the seventh image feature; the third grouped convolution layer is used to: perform grouped convolution processing on the seventh image feature to obtain the eighth image feature; the sixth convolution layer is used to: perform convolution processing on the eighth image feature to obtain the ninth image feature.

[0052] The first Backbone layer is used to perform feature extraction on the target input image to obtain the tenth image feature; the first channel connection layer is used to perform channel connection on the third image feature and the tenth image feature to obtain the eleventh image feature; the second Backbone layer is used to perform feature extraction on the eleventh image feature to obtain the twelfth image feature; the second channel connection layer is used to perform channel connection on the sixth image feature and the twelfth image feature to obtain the thirteenth image feature; the third Backbone layer is used to perform feature extraction on the thirteenth image feature to obtain the fourteenth image feature; the third channel connection layer is used to perform feature extraction on the ninth image feature The first Neck layer is used to perform channel connection on the eleventh image feature, the twelfth image feature and the fourteenth image feature to obtain the fifteenth image feature; the first Head layer is used to perform feature fusion on the eleventh image feature, the twelfth image feature and the fourteenth image feature to obtain the sixteenth image feature; the first Neck layer is used to perform feature fusion on the sixteenth image feature to obtain the first detection result; the second Neck layer is used to perform feature fusion on the thirteenth image feature and the fourteenth image feature to obtain the seventeenth image feature; the second Head layer is used to detect the seventeenth image feature to obtain the second detection result; the third Head layer is used to detect the fifteenth image feature to obtain the third detection result.

[0053] Among them, the first detection result, the second detection result and the third detection result together constitute the road foreign object detection result.

[0054] In an optional manner, the road foreign object detection result is: a target detection frame containing a foreign object in the image to be detected and the probability of the target detection frame.

[0055] Among them, when there are foreign objects in the image to be detected, the number of target detection frames is at least one, and each target detection frame corresponds to a category and probability. Each category of the target detection frame corresponds to a type of foreign object, and the probability of the target detection frame indicates the probability that the target detection frame belongs to a certain category. The larger the probability value, the more realistic the detection result.

[0056] The technical solution of this embodiment combines the long-term and short-term background of the road image and prior information to detect road foreign objects, thereby avoiding the problem of low accuracy caused by collecting a large number of sample images of abnormal road objects. While adapting to road foreign object detection in different complex environmental scenarios, it improves the accuracy of road foreign object detection.

[0057] Figure 3 FIG. 2 is a schematic diagram showing a structure of an embodiment of a road foreign object detection system 200 provided by the present invention. Figure 3 As shown, the system 200 includes: an acquisition module 210 and a detection module 220;

[0058] The acquisition module 210 is used to: acquire an image to be detected that includes a target road area, and extract a long-term background image and a short-term background image corresponding to the image to be detected;

[0059] The detection module 220 is used to obtain a road foreign object detection result of the image to be detected based on the image to be detected, the long-term background image and the short-term background image, and using a trained road foreign object detection model; wherein the road foreign object detection model includes: a convolutional neural network for extracting image prior information and a target detection network for road foreign object detection.

[0060] In an optional manner, the acquisition module 210 is specifically used to:

[0061] The first Gaussian mixture model is used to obtain a long-term background image corresponding to the image to be detected, and the second Gaussian mixture model is used to obtain a short-term background image corresponding to the image to be detected; wherein both the first Gaussian mixture model and the second Gaussian mixture model are trained using sample images containing target road areas, and the update frequency of the first Gaussian mixture model is less than the update frequency of the second Gaussian mixture model.

[0062] In an optional manner, the convolutional neural network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a first grouped convolutional layer, a second grouped convolutional layer, and a third grouped convolutional layer; the target detection network includes: a first Backbone layer, a second Backbone layer, a third Backbone layer, a first channel connection layer, a second channel connection layer, a third channel connection layer, a first Neck layer, a second Neck layer, a first Head layer, a second Head layer, and a third Head layer;

[0063] The first convolution layer is used to: perform convolution processing on the target input image to obtain the first image feature; wherein the target input image is: a nine-channel image composed of the image to be detected, the long-time background image and the short-time background image; the first grouped convolution layer is used to: perform grouped convolution processing on the first image feature to obtain the second image feature; the second convolution layer is used to: perform convolution processing on the second image feature to obtain the third image feature; the third convolution layer is used to: perform convolution processing on the first image feature to obtain the fourth image feature; the second grouped convolution layer is used to: perform grouped convolution processing on the fourth image feature to obtain the fifth image feature; the fourth convolution layer is used to: perform convolution processing on the fifth image feature to obtain the sixth image feature; the fifth convolution layer is used to: perform convolution processing on the fourth image feature to obtain the seventh image feature; the third grouped convolution layer is used to: perform grouped convolution processing on the seventh image feature to obtain the eighth image feature; the sixth convolution layer is used to: perform convolution processing on the eighth image feature to obtain the ninth image feature;

[0064] The first Backbone layer is used to perform feature extraction on the target input image to obtain the tenth image feature; the first channel connection layer is used to perform channel connection on the third image feature and the tenth image feature to obtain the eleventh image feature; the second Backbone layer is used to perform feature extraction on the eleventh image feature to obtain the twelfth image feature; the second channel connection layer is used to perform channel connection on the sixth image feature and the twelfth image feature to obtain the thirteenth image feature; the third Backbone layer is used to perform feature extraction on the thirteenth image feature to obtain the fourteenth image feature; the third channel connection layer is used to perform channel connection on the ninth image feature and the fourteenth image feature to obtain the fifteen image features; the first Neck layer is used to: perform feature fusion on the eleventh image feature, the twelfth image feature and the fourteenth image feature to obtain the sixteenth image feature; the first Head layer is used to: detect the sixteenth image feature to obtain a first detection result; the second Neck layer is used to: perform feature fusion on the thirteenth image feature and the fourteenth image feature to obtain the seventeenth image feature; the second Head layer is used to: detect the seventeenth image feature to obtain a second detection result; the third Head layer is used to: detect the fifteenth image feature to obtain a third detection result; wherein the first detection result, the second detection result and the third detection result together constitute the road foreign object detection result.

[0065] In an optional manner, the road foreign object detection result is: a target detection frame containing a foreign object in the image to be detected and the probability of the target detection frame.

[0066] It should be noted that the beneficial effects of the road foreign object detection system provided by the above embodiment are the same as the beneficial effects of the road foreign object detection method, which will not be described in detail here. In addition, when the system provided by the above embodiment realizes its functions, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

[0067] Among them, the road foreign object detection system of the present invention can be a computer program (including program code) running in a computer device. For example, the road foreign object detection system of the present invention is an application software that can be used to execute the corresponding steps in the road foreign object detection method of the present invention.

[0068] In some embodiments, the road foreign object detection system of the present invention can be implemented by combining software and hardware. As an example, the road foreign object detection system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the road foreign object detection method of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0069] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0070] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned methods for detecting foreign objects on a road is implemented. That is to say, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the method for detecting foreign objects on a road shown in any embodiment of the present invention by calling the computer program.

[0071] In an alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0072] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0073] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 In the figure, only one thick line is used to represent the bus 4002, but this does not mean that there is only one bus or one type of bus.

[0074] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0075] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0076] Among them, the electronic device can also be a terminal device, and the terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0077] It should be noted that Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0078] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods for detecting foreign objects on a road is implemented.

[0079] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0080] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the above-mentioned road foreign object detection method.

[0081] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0082] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0084] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0085] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0086] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0087] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0088] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for detecting foreign objects on a road, characterized in that: include: Acquire an image to be detected that includes a target road area, and extract a long-term background image and a short-term background image corresponding to the image to be detected; According to the image to be detected, the long-term background image and the short-term background image, and using a trained road foreign object detection model, a road foreign object detection result of the image to be detected is obtained; wherein the road foreign object detection model includes: a convolutional neural network for extracting image prior information and a target detection network for road foreign object detection.

2. The method for detecting foreign objects on the road according to claim 1, characterized in that: The step of obtaining the long-term background image and the short-term background image corresponding to the image to be detected comprises: The long-term background image corresponding to the image to be detected is obtained by using a first Gaussian mixture model, and the short-term background image corresponding to the image to be detected is obtained by using a second Gaussian mixture model; wherein both the first Gaussian mixture model and the second Gaussian mixture model are trained using sample images containing the target road area, and the update frequency of the first Gaussian mixture model is less than the update frequency of the second Gaussian mixture model.

3. The method for detecting foreign objects on the road according to claim 1, characterized in that: The convolutional neural network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a first grouped convolutional layer, a second grouped convolutional layer and a third grouped convolutional layer; the target detection network includes: a first Backbone layer, a second Backbone layer, a third Backbone layer, a first channel connection layer, a second channel connection layer, a third channel connection layer, a first Neck layer, a second Neck layer, a first Head layer, a second Head layer and a third Head layer; The first convolution layer is used to perform convolution processing on the target input image to obtain a first image feature; wherein the target input image is a nine-channel image composed of the image to be detected, the long-term background image and the short-term background image; the first grouped convolution layer is used to perform grouped convolution processing on the first image feature to obtain a second image feature; the second convolution layer is used to perform convolution processing on the second image feature to obtain a third image feature; the third convolution layer is used to perform convolution processing on the first image feature to obtain a fourth image feature; the second grouped convolution layer is used to perform grouped convolution processing on the fourth image feature to obtain a fifth image feature; the fourth convolution layer is used to perform convolution processing on the fifth image feature to obtain a sixth image feature; the fifth convolution layer is used to perform convolution processing on the fourth image feature to obtain a seventh image feature; the third grouped convolution layer is used to perform grouped convolution processing on the seventh image feature to obtain an eighth image feature; the sixth convolution layer is used to perform convolution processing on the eighth image feature to obtain a ninth image feature; The first Backbone layer is used to perform feature extraction on the target input image to obtain a tenth image feature; the first channel connection layer is used to perform channel connection on the third image feature and the tenth image feature to obtain an eleventh image feature; the second Backbone layer is used to perform feature extraction on the eleventh image feature to obtain a twelfth image feature; the second channel connection layer is used to perform channel connection on the sixth image feature and the twelfth image feature to obtain a thirteenth image feature; the third Backbone layer is used to perform feature extraction on the thirteenth image feature to obtain a fourteenth image feature; the third channel connection layer is used to perform channel connection on the ninth image feature and the fourteenth image feature to obtain a fifteenth image feature. image features; the first Neck layer is used to: perform feature fusion on the eleventh image feature, the twelfth image feature and the fourteenth image feature to obtain the sixteenth image feature; the first Head layer is used to: detect the sixteenth image feature to obtain a first detection result; the second Neck layer is used to: perform feature fusion on the thirteenth image feature and the fourteenth image feature to obtain the seventeenth image feature; the second Head layer is used to: detect the seventeenth image feature to obtain a second detection result; the third Head layer is used to: detect the fifteenth image feature to obtain a third detection result; wherein the first detection result, the second detection result and the third detection result together constitute the road foreign object detection result.

4. The method for detecting foreign objects on a road according to any one of claims 1 to 3, characterized in that: The road foreign object detection result is: a target detection frame containing foreign objects in the image to be detected and the probability of the target detection frame.

5. A road foreign object detection system, characterized in that: include: Acquisition module and detection module; The acquisition module is used to: acquire an image to be detected that includes a target road area, and extract a long-term background image and a short-term background image corresponding to the image to be detected; The detection module is used to obtain a road foreign object detection result of the image to be detected based on the image to be detected, the long-term background image and the short-term background image, and using a trained road foreign object detection model; wherein the road foreign object detection model includes: a convolutional neural network for extracting image prior information and a target detection network for road foreign object detection.

6. The road foreign object detection system according to claim 5, characterized in that: The acquisition module is specifically used for: The long-term background image corresponding to the image to be detected is obtained by using a first Gaussian mixture model, and the short-term background image corresponding to the image to be detected is obtained by using a second Gaussian mixture model; wherein both the first Gaussian mixture model and the second Gaussian mixture model are trained using sample images containing the target road area, and the update frequency of the first Gaussian mixture model is less than the update frequency of the second Gaussian mixture model.

7. The road foreign object detection system according to claim 5, characterized in that: The convolutional neural network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a first grouped convolutional layer, a second grouped convolutional layer and a third grouped convolutional layer; the target detection network includes: a first Backbone layer, a second Backbone layer, a third Backbone layer, a first channel connection layer, a second channel connection layer, a third channel connection layer, a first Neck layer, a second Neck layer, a first Head layer, a second Head layer and a third Head layer; The first convolution layer is used to perform convolution processing on the target input image to obtain a first image feature; wherein the target input image is a nine-channel image composed of the image to be detected, the long-term background image and the short-term background image; the first grouped convolution layer is used to perform grouped convolution processing on the first image feature to obtain a second image feature; the second convolution layer is used to perform convolution processing on the second image feature to obtain a third image feature; the third convolution layer is used to perform convolution processing on the first image feature to obtain a fourth image feature; the second grouped convolution layer is used to perform grouped convolution processing on the fourth image feature to obtain a fifth image feature; the fourth convolution layer is used to perform convolution processing on the fifth image feature to obtain a sixth image feature; the fifth convolution layer is used to perform convolution processing on the fourth image feature to obtain a seventh image feature; the third grouped convolution layer is used to perform grouped convolution processing on the seventh image feature to obtain an eighth image feature; the sixth convolution layer is used to perform convolution processing on the eighth image feature to obtain a ninth image feature; The first Backbone layer is used to perform feature extraction on the target input image to obtain a tenth image feature; the first channel connection layer is used to perform channel connection on the third image feature and the tenth image feature to obtain an eleventh image feature; the second Backbone layer is used to perform feature extraction on the eleventh image feature to obtain a twelfth image feature; the second channel connection layer is used to perform channel connection on the sixth image feature and the twelfth image feature to obtain a thirteenth image feature; the third Backbone layer is used to perform feature extraction on the thirteenth image feature to obtain a fourteenth image feature; the third channel connection layer is used to perform channel connection on the ninth image feature and the fourteenth image feature to obtain a fifteenth image feature. image features; the first Neck layer is used to: perform feature fusion on the eleventh image feature, the twelfth image feature and the fourteenth image feature to obtain the sixteenth image feature; the first Head layer is used to: detect the sixteenth image feature to obtain a first detection result; the second Neck layer is used to: perform feature fusion on the thirteenth image feature and the fourteenth image feature to obtain the seventeenth image feature; the second Head layer is used to: detect the seventeenth image feature to obtain a second detection result; the third Head layer is used to: detect the fifteenth image feature to obtain a third detection result; wherein the first detection result, the second detection result and the third detection result together constitute the road foreign object detection result.

8. The road foreign object detection system according to any one of claims 5 to 7, characterized in that: The road foreign object detection result is: a target detection frame containing foreign objects in the image to be detected and the probability of the target detection frame.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the road foreign object detection method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the road foreign object detection method as claimed in any one of claims 1 to 4.

Images