Road detection method, device, computing equipment and storage medium

Abstract

A road detection method is disclosed, comprising: acquiring an image pair of a road to be detected, the image pair comprising a first image and a second image captured at different locations; determining optical flow data for the image pair; determining a pixel matching relationship between the first image and the second image based on the optical flow data; and determining damage to the road to be detected based on the pixel matching relationship between the first image and the second image. This road detection method can be used in fields such as mapping, autonomous driving, or smart transportation to determine road damage.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more specifically to a road detection method and apparatus based on optical flow, a computing device, and a computer-readable storage medium. Background Art

[0002] In the process of intelligent road inspection, especially road damage detection, although lidar has higher-precision detection capabilities, its high price and large size seriously hinder its popularization and development.

[0003] With the development of computer vision technology, particularly image processing, intelligent road inspection methods have emerged based on the acquisition (e.g., using binocular cameras) and recognition (e.g., using neural networks) of target images. However, these image-processing-based road inspection methods perform poorly for detecting damage or breakage in the road itself, as they often require binocular cameras to capture images. Traditional binocular camera inspection solutions are sensitive to areas with poor lighting and texture, which can lead to false detections and low detection accuracy. Furthermore, binocular camera detection algorithms are complex and heavily rely on external calibration, resulting in high inspection costs. Summary of the Invention

[0004] To address at least one of the above-mentioned deficiencies, the present disclosure provides a road detection method and apparatus, a computing device, and a computer-readable storage medium, which are primarily based on optical flow estimation technology in computer vision and do not rely on binocular cameras, in order to alleviate, mitigate, or even eliminate some or all of the above-mentioned problems and other possible problems.

[0005] According to one aspect of the present disclosure, a road detection method is proposed, comprising: acquiring an image pair for a road to be detected, the image pair comprising a first image and a second image acquired at different shooting positions; determining optical flow data of the image pair; determining a pixel matching relationship between the first image and the second image based on the optical flow data of the image pair; and determining a damage condition of the road to be detected based on the pixel matching relationship between the first image and the second image.

[0006] In some embodiments, the first image and the second image are respectively acquired synchronously by a first camera and a second camera.

[0007] In some embodiments, determining the damage condition of the road to be detected based on the pixel matching relationship between the first image and the second image includes: determining the spatial coordinates of the point cloud of the actual scene corresponding to the first image based on the pixel matching relationship between the first image and the second image; fitting the surface equation of the road to be detected based on the spatial coordinates of the point cloud of the actual scene; and determining the damage condition of the road to be detected based on the surface equation of the road to be detected and the spatial coordinates of the point cloud of the actual scene.

[0008] In some embodiments, the damage condition of the road to be detected includes at least one of the following: potholes and cracks.

[0009] In some embodiments, determining the damage condition of the road to be detected based on the surface equation of the road to be detected and the spatial coordinates of the point cloud of the actual scene includes: determining at least one of the number, position, area, and depth of potholes on the road to be detected based on the surface equation of the road to be detected and the spatial coordinates of the point cloud of the actual scene; determining at least one of the number, position, width, length, and orientation of cracks on the road to be detected based on the surface equation of the road to be detected and the spatial coordinates of the point cloud of the actual scene.

[0010] In some embodiments, the pixel matching relationship between the first image and the second image is determined based on the optical flow data of the image pair, including: determining the corresponding relationship between the pixels in the first image and the second image based on the optical flow data of the image pair; calculating the matching degree between the corresponding pixels based on the corresponding relationship between the pixels in the first image and the second image; and eliminating pixels below a matching degree threshold from the pixel correspondence in the first image and the second image to obtain the pixel matching relationship between the first image and the second image.

[0011] In some embodiments, determining the optical flow data of the image pair includes: constructing a feature pyramid including n layers of feature maps based on the image pair, n>=2, wherein the 1st layer feature map of the feature pyramid is extracted from the image pair, the kth layer feature map is extracted from the k-1th layer feature map, k is a positive integer greater than 1 and less than or equal to n, and generating the optical flow data of the image pair based on the feature pyramid.

[0012] In some embodiments, generating the optical flow data of the image pair based on the feature pyramid includes: generating the optical flow data of the nth layer feature map based on the nth layer feature map of the feature pyramid and the preset initial optical flow data; calculating the optical flow data of the i-th layer feature map of the feature pyramid in sequence from bottom to top through the following steps, where i is a positive integer greater than or equal to 1 and less than n-1: generating the optical flow data of the i-th layer feature map based on the optical flow data of the i+1th layer feature map and the i-th layer feature map; and generating the optical flow data of the image pair based on the optical flow data of the 1st layer feature map of the feature pyramid.

[0013] In some embodiments, each layer of the feature map of the feature pyramid includes a first feature map and a second feature map corresponding to the first image and the second image respectively, and wherein, generating the optical flow data of the i-th layer feature map based on the optical flow data of the i+1-th layer feature map and the i-th layer feature map includes: performing a view transformation on the second feature map in the i-th layer feature map according to the optical flow data of the i+1-th layer feature map to obtain a transformed second feature map, wherein the view transformation includes upsampling the optical flow data of the i+1-th layer feature map, and performing a first transformation on the second feature map of the i-th layer feature map using the upsampled optical flow data of the i+1-th layer feature map; determining the similarity between the first feature map of the i-th layer feature map and the transformed second feature map; generating the optical flow data of the i-th layer feature map based on the first feature map of the i-th layer feature map, the similarity between the first feature map of the i-th layer feature map and the transformed second feature map, and the optical flow data of the i+1-th layer feature map.

[0014] In some embodiments, the generating of the optical flow data of the image pair based on the optical flow data of the feature map of the first layer of the feature pyramid includes: expanding the receptive field of the optical flow map corresponding to the optical flow data of the feature map of the first layer to generate the optical flow data of the image pair, the receptive field of the optical flow map representing the size of the area mapped by the pixel points in the optical flow map on the first image in the image pair.

[0015] In some embodiments, the view transformation further comprises: performing a mask operation on the second feature map after the first transformation, wherein the mask operation is used to estimate the occlusion status of pixels in the second feature map after the first transformation.

[0016] In some embodiments, determining the similarity between the first feature map of the i-th layer feature map and the transformed second feature map includes: increasing the length and width of the transformed second feature map to obtain an enlarged second feature map; using the first feature map of the i-th layer feature map as a convolution kernel, performing a convolution operation on the enlarged second feature map to obtain the similarity between the first feature map of the i-th layer feature map and the transformed second feature map.

[0017] In some embodiments, determining the optical flow data of the image pair includes determining the optical flow data of the image pair using an optical flow estimation model, wherein the optical flow estimation model is trained by the following steps: obtaining a plurality of sample image pairs for training; inputting the plurality of sample image pairs into the optical flow estimation model to generate optical flow data for each sample image pair; calculating the consistency loss of the sample image pair using the optical flow data of each sample image pair, wherein the consistency loss of the sample image pair includes a weighted sum of a photometric consistency loss and a structural consistency loss between the sample image pairs; and adjusting the parameters of the optical flow estimation model so that the sum of the consistency losses of the plurality of sample image pairs converges.

[0018] In some embodiments, the consistency loss further comprises a smoothness loss of the sample image pair.

[0019] In some embodiments, the feature extractor and the optical flow estimator each comprise one or more convolutional neural networks.

[0020] In some embodiments, the optical flow estimation model includes at least one of the following: PWCnet, Flownet, Fownet2.

[0021] According to another aspect of the present disclosure, a road detection device is proposed, comprising: an acquisition module configured to acquire an image pair for a road to be detected, the image pair comprising a first image and a second image acquired at different shooting positions; a first determination module configured to determine optical flow data of the image pair; a second determination module configured to determine a pixel matching relationship between the first image and the second image based on the optical flow data of the image pair; and a third determination module configured to determine a damage condition of the road to be detected based on the pixel matching relationship between the first image and the second image.

[0022] According to another aspect of the present disclosure, a computing device is proposed, comprising: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor is prompted to perform the steps of the road detection method according to some embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed, the road detection method according to some embodiments of the present disclosure is implemented.

[0024] According to another aspect of the present disclosure, a computer program product is provided, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing at least one step of the road detection method according to some embodiments of the present disclosure.

[0025] In the road detection method and device based on optical flow technology according to some embodiments of the present disclosure, on the one hand, since the use of binocular cameras is not strictly limited during image acquisition, the problem of easy misdetection caused by binocular cameras is avoided, thereby improving the detection accuracy; on the other hand, by using optical flow data to determine the pixel matching relationship, the use of binocular camera algorithms is avoided, thereby significantly simplifying the calculation process and reducing the calculation complexity and detection cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The various aspects, features and advantages of the present disclosure will be readily understood from the following detailed description and accompanying drawings, in which:

[0027] Figure 1 Schematically illustrating an example implementation environment of a road detection method according to some embodiments of the present disclosure;

[0028] Figure 2 Schematically illustrates the road detection method according to some embodiments of the present disclosure. Figure 1 Example interaction flow chart implemented in the example implementation environment;

[0029] Figure 3 A flowchart schematically illustrates a road detection method according to some embodiments of the present disclosure;

[0030] Figure 4A A structural block diagram of an optical flow estimation model according to some embodiments of the present invention is shown;

[0031] Figure 4B Shown Figure 4A The schematic diagram of the feature extractor in the optical flow estimation model according to some embodiments of the present invention is shown;

[0032] Figure 4C Shown Figure 4A The operation flow chart of the optical flow estimator in the optical flow estimation model according to some embodiments of the present invention is shown;

[0033] Figure 4D Schematically illustrating the structure and operating principle of an optical flow estimator of an optical flow estimation model according to some embodiments of the present disclosure;

[0034] Figure 4E A schematic diagram schematically illustrates a principle of a feature transformation component in an optical flow estimator of an optical flow estimation model according to some embodiments of the present disclosure;

[0035] Figure 5A A flowchart of training an optical flow estimation model according to some embodiments of the present disclosure is shown;

[0036] Figure 5B Showing examples of the first image and the second image acquired at different positions;

[0037] Figure 5C An example optical flow map predicted by an optical flow estimation model according to some embodiments of the present disclosure is shown;

[0038] Figure 6 A flowchart schematically illustrates a road detection method according to some embodiments of the present disclosure;

[0039] Figure 7A A flowchart showing a road detection method according to some embodiments of the present disclosure is shown.

[0040] Figure 7B A schematic diagram of the triangulation principle is shown;

[0041] Figure 7C A schematic diagram showing visualization results of a road detection method according to some embodiments of the present disclosure is shown;

[0042] Figure 8 Schematically shows an example block diagram of a road detection device according to some embodiments of the present disclosure;

[0043] Figure 9 An example block diagram of a computing device according to some embodiments of the present disclosure is schematically shown.

[0044] It should be noted that the above drawings are merely schematic and illustrative and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0045] Several embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings so that those skilled in the art can implement the present disclosure. The present disclosure can be embodied in many different forms and for many different purposes and should not be limited to the embodiments described herein. These embodiments are provided to make the present disclosure comprehensive and complete and to fully convey the scope of the present disclosure to those skilled in the art. The embodiments do not limit the present disclosure.

[0046] It will be understood that although the terms first, second, third, etc. may be used to describe various elements, components, and / or parts in this article, these elements, components, and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, or part from another element, component, or part. Therefore, the first element, component, or part discussed below may be referred to as the second element, component, or part without departing from the teachings of the present disclosure.

[0047] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to also include the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "include" and / or "comprise" when used in this specification specify the presence of the features, wholes, steps, operations, elements and / or parts, but do not exclude the presence of one or more other features, wholes, steps, operations, elements, parts and / or their groups or add one or more other features, wholes, steps, operations, elements, parts and / or their groups. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0048] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the relevant art and / or the context of this specification, and will not be interpreted in an idealized or overly formal sense unless expressly defined as such herein.

[0049] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0050] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0051] Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring objects. This involves further processing the images, transforming them into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0052] Before describing the embodiments of the present disclosure in detail, some related concepts are first explained for the sake of clarity:

[0053] 1. Optical flow and optical flow estimation:

[0054] Optical flow is the instantaneous velocity of pixels moving across the observation plane of a moving object. When the time interval is small (such as between consecutive video frames), it is equivalent to the displacement of the target pixel. In computer vision, optical flow can be used to represent the movement of individual pixels in a video image over time.

[0055] Optical flow estimation generally refers to the use of an optical flow estimation algorithm or model (such as a neural network model PWCNet, FlowNet, FlowNet2, etc.) based on two temporally correlated images, such as two adjacent frames in a video, to determine the position of a pixel in the first image (at the time of acquisition) in the second image (at the time of acquisition), and to calculate the motion displacement of the pixel in the time interval between the acquisition times of the two images (i.e., the position offset of the second image relative to the first image). In some embodiments of the present disclosure, optical flow estimation can be expanded to: based on any pair of images (including a first image and a second image, which do not necessarily have temporal correlation) of the same target, use an optical flow estimation algorithm or model to determine the position of a pixel in the first image in the second image, and calculate the position offset of the pixel in the first image relative to the second image.

[0056] 3. Optical flow data of image pairs.

[0057] The positional offset of a pixel in the first image relative to the second image in an image pair determined by optical flow estimation can be referred to as the optical flow of the pixel in the first image relative to the second image. For example, the optical flow of a pixel in the first image relative to the second image can be represented by (u, v), where u is the positional offset of the pixel in the x-direction relative to the second image, and v is the positional offset of the pixel in the y-direction relative to the second image.

[0058] The optical flow data for the first image relative to the second image is defined as the set of optical flows of one or more pixels in the first image relative to the second image. Similarly, the optical flow data for the second image relative to the first image is defined as the set of optical flows of one or more pixels in the second image relative to the first image. Therefore, the optical flow data for an image pair can be defined as either the optical flow data for the first image relative to the second image, or the optical flow data for the second image relative to the first image.

[0059] The optical flow data of a first image relative to a second image (or vice versa) can include dense optical flow data and sparse optical flow data. If the number of pixel optical flows included in the optical flow data of the first image relative to the second image is the same as the total number of pixels in the first image, it can be referred to as dense optical flow data; if the number of pixel optical flows included in the optical flow data of the first image relative to the second image is less than the number of pixels in the first image, it can be referred to as sparse optical flow data. Dense optical flow data for an image can be obtained by calculating the optical flow data for each pixel in the image.

[0060] In road detection methods of related technologies, binocular cameras are usually used to capture road images, and then visual features are matched on the binocular images, which heavily relies on the extrinsic parameter calibration of the binocular camera. However, during the road detection process, the binocular camera installed on the vehicle is easily affected by vehicle bumps and external temperature, causing the calibration extrinsic parameters to change, resulting in serious impact on detection accuracy, or even inability to complete the detection. On the other hand, algorithms based on binocular cameras have a large amount of computational complexity, making it difficult to achieve real-time detection on the vehicle side.

[0061] In response to the above-mentioned problems in the related art, the inventors of the present disclosure proposed using cameras with different shooting positions (not limited to binocular cameras, such as two monocular cameras shooting synchronously or one monocular camera shooting continuously) to obtain a pair of images of the road to be inspected; and innovatively proposed using the optical flow method (using the optical flow estimation model to determine the optical flow data of two images not limited to time correlation) to determine the visual feature matching between the pair of road images, that is, the matching of pixel points in the two images corresponding to the same three-dimensional spatial position in the actual scene; and using the relative shooting positions and orientations of the respective cameras when the two images were taken and the visual feature matching between the images to identify the road to be inspected and its damage condition (for example, using triangulation to determine the depth of the pixel points in the image and estimate the three-dimensional point cloud of the road, and determining the road surface and the damaged area therein based on the point cloud).

[0062] This disclosure also provides a self-supervised optical flow estimation model based on a deep learning neural network for arbitrary image pairs (the two images in the pair do not necessarily need to be temporally correlated). This training process requires only unlabeled training samples (i.e., image pairs, without corresponding ground-truth optical flow values). Model training (optimization of model parameters) is achieved through a self-supervisory signal based on a consistency loss. This avoids the difficult-to-obtain ground-truth optical flow values required for training in traditional supervised optical flow estimation models.

[0063] Figure 1 The following schematically shows an example implementation environment 100 of a road detection method according to some embodiments of the present disclosure. Figure 1As shown, implementation environment 100 may include a terminal device 110, a server 120, and a network 130 for connecting terminal device 110 and server 120. In some embodiments, terminal device 110 may be used to implement the road detection method according to the present disclosure. For example, terminal device 110 may be deployed with corresponding programs or instructions for executing the various methods provided by the present disclosure. Optionally, server 120 and terminal device 110 may also cooperate with each other to implement the various methods according to the present disclosure.

[0064] The terminal device 110 can be any type of mobile computing device, including a mobile computer (e.g., a Microsoft® Surface® device, a personal digital assistant (PDA), a laptop computer, a notebook computer, a tablet computer such as an Apple iPad™, a netbook, etc.), a mobile phone (e.g., a cellular phone, a smartphone such as a Microsoft Windows® phone, an Apple iPhone, a phone that implements the Google® Android™ operating system, a Palm® device, a Blackberry® device, etc.), a wearable computing device (e.g., a smart watch, a head-mounted device including smart glasses such as Google® Glass™, etc.), or other types of mobile devices (e.g., Figure 1 In some embodiments, the terminal device 110 may also be a stationary computing device, such as a desktop computer, a game console, or a smart TV.

[0065] The server 120 may be a single server or a server cluster, or may be a cloud server or cloud server cluster that can provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It should be understood that the servers mentioned herein are typically server computers with large amounts of memory and processor resources, but other embodiments are also possible. Alternatively, the server 120 may also be an ordinary desktop computer, which includes a host, a display, and the like.

[0066] Examples of network 130 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet. Server 120 and terminal device 110 may include at least one communication interface (not shown) capable of communicating via network 130. Such a communication interface may be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), a wired or wireless wireless interface (such as IEEE 802.11 wireless LAN (WLAN)), a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth™ interface, a Near Field Communication (NFC) interface, and the like.

[0067] like Figure 1 As shown, the terminal device 110 may include a display screen, and the terminal user may interact with the terminal application via the display screen. The terminal device 110 may communicate with the server 120, for example, via the network 130, for example, to send data to or receive data from the server 120. The terminal application or game may be a local application, a web application, or a mini-program (LiteApp, such as a mobile app or WeChat app) that is a lightweight application. If the terminal application is a local application that requires installation, the terminal application may be installed on the terminal device 110. If the terminal application is a web application, the terminal application or game may be accessed via a browser. If the terminal application is a mini-program, the terminal application may be opened directly on the terminal device 110 by searching for relevant information about the terminal application (such as the name of the terminal application) or scanning a graphic code (such as a barcode or QR code) of the terminal application, without having to install the terminal application.

[0068] Figure 2 The road detection method according to some embodiments of the present disclosure is shown in FIG. Figure 1 The example interactive flow chart implemented in the example implementation environment 100 is shown below. Figure 2 The example interaction flowchart shown briefly describes the working principle of the road detection method according to some embodiments of the present disclosure in the implementation environment 100.

[0069] like Figure 2As shown, first, the server 120 can be configured to: obtain an image pair for the road to be detected from the terminal device 110, and the image pair includes a first image and a second image obtained at different shooting positions. For example, the terminal device 110 can obtain the image of the road to be detected through a built-in image acquisition device (such as a camera or a camera, etc.). Secondly, the server 120 can be configured to: input the image pair into the optical flow estimation model to generate optical flow data of the image pair. Thirdly, the server 120 can be configured to: determine the pixel matching relationship between the first image and the second image based on the optical flow data of the image pair. Finally, the server 120 can be configured to: determine the damage condition of the road to be detected based on the pixel matching relationship between the first image and the second image, and send the damage condition of the road to be detected to the terminal device 110 so that the user of the terminal device 110 can be informed of the condition.

[0070] Figure 1 and Figure 2 The implementation environment and interaction flow diagram are merely illustrative, and the road detection method according to the present disclosure is not limited to the exemplary implementation environment and interaction flow shown. It should be understood that although the server 120 and the terminal device 110 are shown and described herein as separate structures, they can be different components of the same computing device.

[0071] Optionally, all steps of the road detection method according to some embodiments of the present disclosure may also be implemented only at the terminal device 110, that is, the terminal device 110 may be configured to: obtain an image pair for the road to be detected, the image pair including a first image and a second image obtained at different shooting positions; input the image pair into an optical flow estimation model to generate optical flow data of the image pair; determine the pixel matching relationship between the first image and the second image based on the optical flow data of the image pair; determine the damage condition of the road to be detected based on the pixel matching relationship between the first image and the second image.

[0072] Figure 3 A flow chart of a road detection method according to some embodiments of the present disclosure is schematically shown.

[0073] In some embodiments, the road detection method according to the present disclosure can be used in, for example, Figure 1 and Figure 2 In other embodiments, the road detection method according to the present disclosure may also be executed on the terminal device 110 (for example, an in-vehicle intelligent navigation system or an autonomous driving system). Figure 1 and Figure 2 The terminal device 110 and the server 120 shown are implemented in combination.

[0074] like Figure 3As shown, the road detection method according to some embodiments of the present disclosure may include steps S310-S330.

[0075] In step S310 , an image pair for a road to be detected is acquired, wherein the image pair includes a first image and a second image acquired at different shooting positions.

[0076] According to the concept of the present disclosure described above, in order to avoid the tedious calculations and low precision of road recognition using a single or binocular camera, dual monocular cameras can be used to obtain two images of the same section of the road to be detected, and then the optical flow method and geometric optics methods (such as triangulation) can be combined to identify road damage.

[0077] In some embodiments of the present disclosure, in step 310, two different monocular cameras, such as a first camera and a second camera, located at different positions can be used to synchronously capture the first and second images. For example, the first and second cameras can be fixedly mounted at different positions on the vehicle (or they can be removably mounted, as long as they are in different positions during capture), such as side-by-side horizontally or vertically (left-right or top-bottom), to ensure that the two images have a certain difference, which is beneficial for subsequent optical flow estimation and road recognition (especially triangulation of pixel depth). The so-called "synchronous" capture means that the two cameras capture the first and second images simultaneously or nearly simultaneously (i.e., the interval between the capture times is very small, for example, less than a certain threshold, such as 10 milliseconds or less), to ensure that the content of the two images is substantially consistent, or the difference is not too significant. Otherwise (due to changes in the captured content caused by vehicle movement), it may be impossible to find matching pixels.

[0078] Alternatively, the first camera and the second camera may be the same monocular camera, with the first image and the second image being two consecutive frames captured by the same camera, with a short interval between captures to ensure that the image content is substantially consistent. In some embodiments, the first camera and the second camera may also be binocular cameras. This disclosure does not limit the camera type, as long as two images of the same target road to be inspected can be obtained.

[0079] In some embodiments, the first image and the second image of the road to be detected include not only the road to be detected itself, but also vehicles (cars, bicycles, etc.) and pedestrians on the road, as well as the environment around the road, such as buildings, plants, sky, overpasses, etc. Figure 5B shown.

[0080] At step S320 , optical flow data of the image pair is determined.

[0081] According to the concepts of the present disclosure, optical flow methods can be used to obtain pixel matching relationships between the first and second images in an image pair. In some embodiments, optical flow data for the image pair can be directly obtained using a trained neural network-based optical flow estimation model. Such optical flow estimation models may include, for example, PWCNet, FlowNet, and FlowNet2.

[0082] In some embodiments, the optical flow estimation model can also be trained using road-related optical flow training samples, that is, the model parameters can be optimized to improve the accuracy of model predictions. For example, the neural network-based optical flow estimation model can be represented as Mof, where the model parameter is θ (for example, it can be a parameter vector composed of multiple parameters). The optical flow estimation model is then processed on an input image pair (including a first image I1 and a second image I2), and the resulting optical flow data for the image pair can be represented as Mof(θ, I1, I2). The optical flow prediction data Mof(θ, I1, I2) can represent the optical flow data of I1 relative to I2, or the optical flow data of I2 relative to I1.

[0083] Suppose, for pixel A(x1, y1) in image I1, the optical flow estimation model uses (for example, one or more convolutional neural networks) to find a corresponding or matching pixel B(x2, y2) in image I2. From this, the positional offset or displacement of pixel A relative to image I2 can be calculated as (u, v) = (x2, y2) - (x1, y1), where u is the displacement in the x-direction and v is the displacement in the y-direction. The more accurately pixel B is determined, the more accurate the displacement is, and the more accurate the generated optical flow prediction data. Here, "correspondence or matching" between pixels A and B can mean that the spatial locations corresponding to these two pixels in the actual scene are consistent or identical, that is, points A and B are the projections of the same location (three-dimensional coordinate point) in the actual scene in the two images. It should be noted that the pixel matching relationship reflected in the optical flow data is not necessarily a perfect or exact match, because the optical flow data is predicted and estimated by the model and is not necessarily the true optical flow value.

[0084] To improve the accuracy of optical flow estimation models, especially traditional supervised models, training requires building a large database of labeled training samples containing image pairs and their corresponding ground-truth optical flow data (as labels). However, in practice, obtaining ground-truth optical flow data for images is difficult, meaning that labeling image pairs is challenging.

[0085] In some embodiments, in view of the above situation, in addition to the optical flow estimation model described above, the inventors have proposed a self-supervised optical flow estimation model for estimating the optical flow data of the image pair in step 320. The self-supervised optical flow estimation model according to some embodiments of the present invention is a self-supervised optical flow estimation model based on a deep learning neural network. Its training process only requires the input of unlabeled training samples (i.e., image pairs, without the corresponding optical flow truth values), and the model training (optimization of model parameters) is achieved through a self-supervised signal based on consistency loss. In this way, the difficult-to-obtain optical flow truth values required for training in traditional supervised optical flow estimation models are avoided. For the specific structure and principle of this self-supervised optical flow estimation model, please refer to Figures 4A-4D and its corresponding description

[0086] In step S330 , a pixel matching relationship between the first image and the second image is determined based on the optical flow data of the image pair.

[0087] As described above, after determining the optical flow data of the image pair, the optical flow, i.e., the displacement, of the pixels in the first image of the image pair relative to the second image can be obtained, so that a pixel matching relationship between the first image and the second image can be established for subsequent road recognition.

[0088] For example, for an image pair, including images I1 and I2, the optical flow data obtained by the optical flow estimation model shows that the optical flow (i.e., displacement) of pixel point A (x1, y1) in image I1 relative to image I2 is (u, v), then the coordinates of pixel point B in image I2 corresponding to or matching pixel point A in image I1 can be obtained as (x2, y2) = (x1, y1) + (u, v).

[0089] In some embodiments, due to errors in the optical flow estimation model training process and other factors (such as temporary occlusion of the road in the image), the optical flow data obtained in step S320 may contain inaccurate pixel optical flow estimates, and thus inaccurate pixel matching or correspondence. Therefore, after obtaining the first pixel matching relationship based on the optical flow data, an additional step can be added to eliminate possible mismatched pixels to obtain the final pixel matching relationship. For the specific process of mismatch elimination, please refer to Figure 6 and its related descriptions.

[0090] In step S340, the damage condition of the road to be inspected is determined based on the pixel matching relationship between the first image and the second image.

[0091] After obtaining the pixel matching relationship in the image pair, we can use triangulation to calculate the depth of the 3D spatial point (represented by 3D coordinates) in the actual scene corresponding to the pixel in the first image (i.e., the distance from the spatial point to the optical center of the first camera) based on the pixel matching relationship and the pre-calibrated intrinsic parameters of the first and second cameras. The principle of triangulation is: when observing the same 3D point P(x, y, z) at different locations, and knowing the 2D projection points X1(x1, y1) and X2(x2, y2) of the 3D point observed at different locations, the depth information of the 3D point can be recovered using triangulation.

[0092] After obtaining the depth of each 3D spatial point in the actual road scene to be inspected, geometric methods can be used to calculate the 3D coordinates of each location or spatial point in the actual scene corresponding to each pixel in the first or second image based on this depth information. This is the coordinates of a point cloud consisting of multiple points representing the road to be inspected and its surroundings. Subsequently, using the point cloud coordinates and a surface fitting algorithm, the surface equation of the road to be inspected can be fitted. Finally, the road damage status is determined by finding points in the point cloud that lie below the road surface.

[0093] For example, if a number of points located below the surface appear in a certain area (more than a threshold number of points appear below the surface in a small area), then the area may be a damaged area, especially a pothole area. In some embodiments of the present disclosure, the damage to the road may include at least one of potholes and cracks. Potholes appear as depressions relative to the road surface due to road damage, while cracks appear as straight or curved gaps in the road due to aging or quality problems. The pothole situation can be described by the following parameters: center point position, area, depth (deepest point), etc.; the crack situation can be characterized by the following parameters: crack length, depth, width, orientation, etc. The various parameters of the above damage conditions can be obtained using geometric methods through point cloud coordinates and surface equations.

[0094] According to some embodiments of the present disclosure, the road detection method uses a camera (not limited to a binocular camera) to capture two images (i.e., an image pair) of the same road to be detected, and uses the optical flow method to determine the pixel matching relationship between the two images, and then identifies the damage condition of the road to be detected based on the pixel matching relationship. Therefore, in the road detection method based on optical flow technology according to some embodiments of the present disclosure, on the one hand, because the image acquisition is not strictly limited to the use of binocular cameras, the detection accuracy is improved; on the other hand, by using optical flow data to determine the pixel matching relationship, the defects of the binocular camera-based algorithm are avoided, thereby significantly simplifying the calculation process and reducing the detection and camera maintenance costs. In addition, the road detection method according to the embodiments of the present disclosure can be applied to determine the damage condition of roads in fields such as mapping, autonomous driving, or smart transportation.

[0095] Figure 4A A structural block diagram of an optical flow estimation model according to some embodiments of the present invention is shown.

[0096] As mentioned above, in Figure 3 In the road detection method according to some embodiments of the present invention, the optical flow estimation model in step S320 can use a model of related technology that has been trained, so the required optical flow data can be obtained directly by inputting the image pair of the road to be detected into the model. In addition, the road detection method according to the present disclosure can also include step 320: using Figure 4A The optical flow estimation model shown in some embodiments of the present disclosure is used to obtain optical flow data of an image pair.

[0097] like Figure 4A As shown, the optical flow estimation model 400 according to some embodiments of the present disclosure may include a feature extractor 410 and an optical flow estimator 420. In some embodiments, either the feature extractor 410 or the optical flow estimator 420 may be composed of one or more convolutional neural networks. Figure 4A As shown, the acquired image pair, i.e., the first image I1 and the second image I2, is processed in sequence by the feature extractor 410 (extracting features) and the optical flow estimator 420 (estimating optical flow) in the optical flow estimation model to obtain the optical flow data F of the image pair.

[0098] More specifically, Figure 3 The step S320-determining the optical flow data of the image pair may include: constructing a feature pyramid comprising n layers of feature maps based on the image pair, where n>=2, wherein the first layer of the feature pyramid is extracted from the image pair, the kth layer of the feature map is extracted from the k-1th layer of the feature map, and k is a positive integer greater than 1 and less than or equal to n; and generating the optical flow data of the image pair based on the feature pyramid. Figure 4B and 4C As shown, the above two steps can be completed by the feature extractor 410 and the optical flow estimator 420 respectively.

[0099] A feature pyramid consists of multiple layers of feature maps extracted progressively from the original image, with their resolution and scale decreasing as the number of layers increases. Deep feature maps, while having low resolution, have a larger receptive field, making them suitable for detecting complex targets. Shallow feature maps, on the other hand, have high resolution but a smaller receptive field, lacking detailed information, and are therefore suitable for detecting small targets. Therefore, because the optical flow data determination method according to the present disclosure is based on a feature pyramid, it can fuse information from multiple layers (including deep and shallow feature maps). This allows for the separation of simple targets from shallow ones, while complex ones from shallow ones, for targets of varying sizes and complexities in the original image. Therefore, the feature pyramid-based optical flow estimation method according to some embodiments of the present disclosure can more accurately reflect information about various types of targets (simple or complex) in the original image, resulting in higher-precision optical flow data.

[0100] Figure 4B Shown Figure 4A FIG. 1 is a schematic diagram of a feature extractor in an optical flow estimation model according to some embodiments of the present invention.

[0101] like Figure 4B As shown, feature extractor 410 can be configured to extract features layer by layer based on the acquired image pair, i.e., first image I1 and second image I2 (e.g., perform layer-by-layer feature extraction on the first image and the second image, respectively), to construct a feature pyramid comprising n layers of feature maps, where n >= 2. n can be considered a hyperparameter and is predetermined based on specific application scenarios. In some embodiments, each layer of feature maps includes a first feature map and a second feature map corresponding to the first image and the second image, respectively. Thus, each layer of the feature pyramid can be a feature map pair consisting of a first feature map and a second feature map.

[0102] The so-called "layer-by-layer feature extraction" means: first extract features from the original images I1 and I2 respectively to obtain the first layer feature map (or feature map pair), that is, Figure 4B The two feature maps marked as "1" on the left of the feature pyramid (the upper and lower feature maps marked as "1" represent the first-layer feature maps of I1 and I2 respectively); then, features are extracted from the two feature maps in the first-layer feature map pair to obtain the second-layer feature map pair, that is, Figure 4B The two feature maps marked as "two" on the left in the feature pyramid (the upper and lower feature maps marked as "two" represent the first-layer feature maps of I1 and I2 respectively); and so on, until the nth layer is obtained, Figure 4B In this way, by extracting features layer by layer by the feature extractor 410, n-layer features of the image pair (i.e., the first image I1 and the second image I2) are obtained, i.e., n-layer feature map pairs, for example Figure 4BThe 6 layers of feature map pairs shown, namely the feature map pairs marked as “1” to “6”, respectively represent the first to sixth layers of feature map pairs of the feature pyramid. Figure 4B As shown, in the 6-layer feature pyramid, the upper pyramid is the feature pyramid of the first image I1, and the lower pyramid is the feature pyramid of the second image I2.

[0103] In some embodiments, the feature extractor 410 may include a multi-layer convolutional neural network for extracting feature maps at each layer of the feature pyramid. The feature maps obtained by processing different convolutional neural networks at each layer have different sizes, where the size of the feature map represents the number of pixels (which correspond to one or more pixels of the original image) included in the feature map. Generally, after feature extraction is performed by convolution, the size of the feature map gradually decreases, i.e. Figure 4B The feature maps from the first to the sixth layer shown in the figure gradually decrease, while the number of channels gradually increases ( Figure 4B Channel sizes are not shown in the figure. In some embodiments, the number of channels in each layer of the feature pyramid can be 16, 32, 64, 96, 128, or 196, respectively. In some embodiments, during feature extraction using convolution operations, the first image I1 and the second image I2 can share weights (i.e., undergo the same multi-layer convolution operations) so that the difference between the two feature maps in each layer after feature extraction is not too large, thereby facilitating subsequent optical flow estimation operations.

[0104] Figure 4C Shown Figure 4A FIG. 1 is a flowchart showing the operation of an optical flow estimator in an optical flow estimation model according to some embodiments of the present invention.

[0105] like Figure 4C As shown, the optical flow estimator 420 can be configured to generate optical flow data for the image pair based on a feature pyramid. In some embodiments, generating optical flow data for the image pair based on a feature pyramid includes:

[0106] Generate optical flow data of the n-th layer feature map according to the n-th layer feature map of the feature pyramid and the preset initial optical flow data;

[0107] The optical flow data of the feature map of the i-th layer of the feature pyramid is calculated sequentially from bottom to top by the following steps, where i is an integer greater than or equal to 1 and less than or equal to n-1: the optical flow data of the feature map of the i+1-th layer is generated based on the optical flow data of the feature map of the i-th layer and the feature map of the i-th layer; and

[0108] Based on the optical flow data of the feature map of the first layer of the feature pyramid, the optical flow data of the image pair is generated.

[0109] The above steps can be performed using the optical flow estimator 420 by Figure 4C Steps S401-S404 are completed. In the above-mentioned process of cyclically calculating the optical flow data of each layer of feature maps based on the feature pyramid, except for the nth layer (i.e., the bottom layer), the optical flow data of each layer is the result of fine-tuning the optical flow data of the feature map of the previous layer based on the feature map of the current layer. This cyclic operation can fully utilize the information of the feature map of each layer to gradually improve the accuracy of the optical flow data, thereby more effectively obtaining high-precision optical flow information.

[0110] like Figure 4C As shown, in some embodiments, the optical flow estimator 420 may be further configured to perform the following steps:

[0111] S401, generating optical flow data of the n-th layer feature map according to the n-th layer feature map and preset initial optical flow data, and setting a counter k=n-1;

[0112] S402, determine the loop end condition, whether k is greater than zero, if yes, go to step S403 to continue the optical flow estimation of the current layer (i.e., the kth layer), otherwise go to step S404 and end the loop;

[0113] S403, generating optical flow data of the k-th layer feature map according to the optical flow data of the k+1-th layer feature map and the k-th layer feature map, and setting k=k-1;

[0114] S404: Generate optical flow data of the image pair based on the optical flow data of the first layer feature map.

[0115] according to Figure 4C As shown, in the optical flow estimation model 400 according to some embodiments of the present disclosure, the optical flow estimator 420 calculates the corresponding optical flow data (which can be called the optical flow data of the feature map of this layer) for each layer of feature maps or feature map pairs from the bottom up of the n-layer feature pyramid from the feature extractor 410, until the corresponding optical flow data of the first layer of feature maps or feature map pairs (i.e., the optical flow data of the first layer of feature maps) is obtained; finally, the first layer of optical flow data can be used as the final optical flow data of the original input image pair (i.e., the first image I1 and the second image I2). In the above-mentioned process of cyclically calculating the optical flow data, except for the nth layer (i.e., the bottom layer), the optical flow data of each layer is the result of fine-tuning the optical flow data of the feature map of the previous layer obtained based on the feature map of this layer. Such an operation can more effectively detect high-precision optical flow information.

[0116] In some embodiments, Figure 4CThe step S404 - generating the optical flow data of the k-th layer feature map according to the optical flow data of the k+1-th layer feature map and the k-th layer feature map, includes: performing a view transformation on the second feature map in the k-th layer feature map according to the optical flow data of the k+1-th layer feature map to obtain a transformed second feature map, wherein the view transformation includes upsampling the optical flow data of the k+1-th layer feature map, and performing a first transformation on the second feature map of the k-th layer feature map using the upsampled optical flow data of the k+1-th layer feature map; determining the similarity between the first feature map of the k-th layer feature map and the transformed second feature map; generating the optical flow data of the k-th layer feature map based on the first feature map of the k-th layer feature map, the similarity between the first feature map of the k-th layer feature map and the transformed second feature map, and the optical flow data of the k+1-th layer feature map. The above steps can be used Figure 4D The various components of the optical flow estimator 420 are implemented as shown.

[0117] In the above embodiment, the optical flow data for each layer of feature maps is generated through three steps: feature transformation, similarity determination, and optical flow estimation. In the feature transformation step, the second feature map is implemented based on the previously calculated optical flow data of the previous layer. This reduces the difference between the second feature map and the first feature map, compensates for geometric distortion or deformation, and thus reduces the cost of subsequent similarity calculations. The similarity determination step obtains the similarity between the first feature map and the second feature map for subsequent optical flow estimation. This ensures that the final optical flow data of the feature map of the current layer incorporates the similarity weights between the corresponding pixels of the first and second feature maps, resulting in more accurate calculation results. The optical flow estimation step integrates three inputs: the first feature map of the current layer, the similarity, and the optical flow data of the feature map of the previous layer. Through multi-layer convolution operations, the optical flow data of the feature map of the current layer is ultimately obtained. Therefore, the optical flow estimation operation not only reflects the previous results (the optical flow data of the previous layer) but also incorporates richer feature information (the similarity weights and the feature map of the current layer), further improving the accuracy of the optical flow estimation results.

[0118] Figure 4D The structure and operation principle diagram of the optical flow estimator 420 of the optical flow estimation model 400 according to some embodiments of the present disclosure are schematically shown. Figure 4C The operation process of the optical flow estimator 420 shown can be Figure 4D The individual components shown are implemented.

[0119] like Figure 4D As shown, the optical flow estimator 420 may include a feature transformation component 421, a similarity calculation component 422, and an optical flow estimation component 423. Optionally, the optical flow estimator may further include a receptive field expansion component 424. For illustrative purposes, Figure 4D Only shown Figure 4BThe processing process of a feature map layer in the feature pyramid in the optical flow estimator 420 is shown.

[0120] The following combination Figure 4D , which details the specific processing of the feature map of each layer in the feature pyramid by the optical flow estimator 420 in the optical flow estimation model 400 according to some embodiments of the present invention, and the final generation process of the optical flow data of the image.

[0121] First, if Figure 4D As shown, the optical flow data of the current layer (i.e., the i-th layer (i=n-1,…,1)) of the second image I2, the feature map, and the previously estimated feature map of the previous layer (i.e., the i+1-th layer) are input into the feature transformation component 421. The feature transformation component 421 is configured to: perform a view transformation on the i-th layer feature map (i.e., the second feature map of the i-th layer feature map) of the second image I2 according to the optical flow data of the i+1-th layer feature map to obtain a transformed second feature map, wherein the view transformation includes upsampling the i+1-th layer optical flow data and performing a first transformation on the second feature map of the i+1-th layer feature map using the optical flow data of the upsampled i+1-th layer feature map. Optionally, the view transformation further includes: performing a mask operation on the second feature map after the first transformation, wherein the mask operation is used to estimate the occlusion of pixels in the second feature map after the first transformation. This step can be completed using the feature transformation component 421, see below in conjunction with Figure 4E In the description

[0122] As mentioned above, generally, Figure 4B As shown in the figure, due to the layer-by-layer feature extraction operation, the size of the feature map of the i+1th layer in the feature pyramid is smaller than the size of the feature map of the i-th layer. Therefore, the size of the optical flow data of the i+1th layer obtained based on the feature map of the i+1th layer is also smaller than the size of the feature map of the i-th layer. Therefore, in order to achieve scale consistency and realize the feature transformation operation, the size of the optical flow data of the i+1th layer can be made equal to the size of the feature map of the i-th layer through upsampling.

[0123] In some embodiments, a view transformation operation may be performed on the second feature map (corresponding to the second image I2) in the feature map of the i-th layer according to the guidance of the optical flow data of the i+1-th layer according to the following formula (1):

[0124]

[0125] Where x is the pixel point of the second feature map in the i-th layer feature map, (x) is the feature vector of the second feature map at pixel x, (x) is the feature vector of the second feature map at pixel x after transformation, ( )(x)represents the optical flow at pixel x in the optical flow data of the i+1th layer feature map, and up refers to the upsampling (e.g., interpolation) operation on the optical flow data of the i+1th layer feature map.

[0126] In some embodiments, the view map transformation operation shown in formula (1) can be implemented by an interpolation algorithm, which can include, for example, bilinear interpolation or other interpolation algorithms.

[0127] Please note that Figure 4D In the case of i=n, since there is no n+1th layer feature map in the feature pyramid, it is necessary to set an initial value of optical flow data as the optical flow data of the n+1th layer feature map, that is, the preset initial optical flow data. For example, the initial optical flow data can be set to zero, that is, ( )(x)=0,up( )(x)=0, to facilitate subsequent calculations.

[0128] Figure 4E A schematic diagram showing the principle of the feature transformation component 421 is shown.

[0129] like Figure 4E As shown, the feature transformation component 421 can be further configured to: perform a mask operation on the second feature map after the first transformation to obtain the transformed second feature map, wherein the mask operation is used to estimate the occlusion of the pixel points in the second feature map after the first transformation.

[0130] like Figure 4E As shown, the masking operation occludes each feature element of the feature map to varying degrees by adjusting the mask matrix Φ and parameters θ and μ. This ensures that the ratio of the value of each feature element in the feature map after occlusion to the value before occlusion is between 0 and 1. Assuming that the second feature map after view transformation has a size of w*h and c channels, the mask matrix Φ has the same size as the second feature map, also w*h, where each element has a value between 0 and 1, representing the probability that the corresponding pixel in the feature map is occluded.

[0131] In some embodiments, the mask matrix Φ and parameters θ and μ are generated in real time by a mask generation network based on the input feature map. The parameters of the mask generation network are trained and obtained during the training of the optical flow estimation model. Alternatively, the mask generation network can be a convolutional neural network, a fully connected network, or the like. The mask network structure disclosed herein can estimate pixels in an image that are occluded by the motion of dynamic objects, resolving the issue of mismatched pixels impacting the proper convergence of the loss function of the optical flow estimation model.

[0132] Secondly, if Figure 4DAs shown, the output result of the feature transformation component 421, i.e., the transformed second feature map of the i-th layer feature map, and the first feature map of the i-th layer feature map are input to the similarity calculation component 422 for processing. The similarity calculation component 422 is configured to determine the similarity between the first feature map of the i-th layer feature map and the transformed second feature map, which can optionally be simply referred to as the similarity of the i-th layer feature map.

[0133] In some embodiments, a convolution operation can be performed on the first feature map (corresponding to the first image I1) and the transformed second feature map (corresponding to the second image I2) in the feature map of the i-th layer according to the following formula (2) to calculate the similarity between the two:

[0134] (2)

[0135] Where x1 and x2 are the pixels of the first feature map and the transformed second feature map respectively. is the similarity of the feature map of the i-th layer, N is the vector The length of T is the transpose operation of the vector, are the feature vector of pixel x1 in the first feature map of the i-th layer feature map and the feature vector of pixel x2 in the second feature map after transformation.

[0136] In some embodiments, the similarity calculation component 422 is further configured to increase the length and width of the transformed second feature map to obtain an enlarged second feature map; and perform a volume convolution operation on the enlarged second feature map using the first feature map as a convolution kernel to obtain the similarity of the feature map of the i-th layer. Optionally, in order to obtain sufficient similarities between different feature combinations of the first feature map and the transformed second feature map in the i-th layer, the size of the transformed second feature map can be increased, and then a sliding window similarity volume convolution operation (such as that of formula (2)) can be performed on the first feature map (i.e., using the first feature map as a convolution kernel and performing a volume convolution operation on the enlarged second feature map). The result obtained from each sliding window volume convolution operation can be used as a feature channel layer, so that similarities including multiple channels can be obtained. For example, the length and width of the changed second feature map can be increased by 8 pixels each, and then the first feature map is used to perform a sliding window volume convolution operation on the enlarged second feature map according to formula (2). This operation can be performed 81 times to obtain 81 similarities, that is, there are 81 channels in total, that is, the total size of the convolution after the operation is 81*w*h, where w and h are the width and height of the first feature map respectively.

[0137] Again, as Figure 4DAs shown, after similarity estimation, the result of the similarity calculation component 422 and the optical flow data of the feature map of the previous layer (i+1 layer) can be input into the optical flow estimation component 423 for processing to obtain the optical flow data of the feature map of the current layer (i-th layer). The optical flow estimation component 423 is configured to generate the optical flow data of the feature map of the i-th layer based on the first feature map in the feature map of the i-th layer, the similarity of the feature map of the i-th layer, and the optical flow data of the feature map of the i+1-th layer.

[0138] In some embodiments, the optical flow estimation component 423 may include one or more convolutional neural networks, such as DenseNet. Specifically, the optical flow estimation component 423 generates the optical flow data of the current layer (layer i) by first merging the first feature map of the feature map of the current layer (layer i), the similarity of the feature map of the current layer, and the optical flow data of the upsampled feature map of the previous layer (layer i+1) together (i.e., superimposing them together as a whole); then, the merged features are subjected to multiple layers of convolution operations (e.g., 4 layers of DenseNet convolution) to finally obtain the optical flow data of the feature map of the current layer (layer i).

[0139] Finally, if Figure 4D As shown, for the optical flow data of the current layer (i-th layer) obtained by the optical flow estimation component 423, different processing can be performed according to the value of i. Figure 4D As shown, when i is not equal to 1, it means that the current layer processed by the optical flow estimator 420 is not the last layer of the feature pyramid. Therefore, the obtained optical flow data of the current i-th layer needs to be re-input into the feature transformer 421 to participate in the estimation process of the optical flow data of the next layer. This cycle continues until the optical flow data of the feature map of the first layer (the last layer to be processed) of the feature pyramid is finally obtained. In other words, when i=1, it means that the current layer processed by the optical flow estimator 420 is the top layer (the last layer to be processed) of the feature pyramid, that is, the first layer. Therefore, the optical flow data of the current layer (i.e., the first layer) obtained can be considered as the optical flow data of the final original image pair (I1 and I2).

[0140] In some embodiments, optionally, as Figure 4D As shown, when i=1, that is, when the current layer is the first layer, the obtained first layer optical flow data can be input into the optional receptive field expansion component 424 to obtain the final image pair optical flow data. The receptive field expansion component can be configured to expand the receptive field of the optical flow map corresponding to the optical flow data of the first layer feature map to generate the optical flow data of the image pair. The receptive field of the optical flow map can represent the size of the area mapped by the pixel points in the optical flow map on the first image in the image pair.

[0141] Generally speaking, the receptive field is the size of the area on the original image that is mapped to the pixels in the feature map output by each layer of a convolutional neural network. This "mapping" refers to the correspondence between pixels in the feature map, obtained by extracting features from the original image through convolution, and an area in the original image. For example, if an original image undergoes one or more convolution operations, and a pixel in the extracted feature map corresponds to (i.e., is extracted from) four pixels in the original image, then the area on the original image where the pixel in that feature map is mapped to four pixels is the receptive field of that feature map.

[0142] In some embodiments, the optical flow data of the first-layer feature map obtained by the above-mentioned feature pyramid hierarchical optical flow estimation method can also be regarded as features extracted based on the original image. Therefore, the receptive field of the optical flow map corresponding to the optical flow data is the size of the area mapped by the pixels in the optical flow map on the first image in the original image pair. To expand the receptive field, in some embodiments of the present invention, a receptive field expansion component 424 is added to the optical flow estimator 420, which is used to add a receptive field expansion operation to the optical flow data of the last layer of feature maps from the bottom to the top of the feature pyramid (i.e., the optical flow data of the first-layer feature maps) to refine the optical flow data and add more detailed information, thereby improving the optical flow data. In some embodiments, the receptive field expansion component 424 can be composed of a multi-layer convolutional neural network, each layer of which can use a dilated convolution operation, and the dilated coefficients of the dilated convolutions of each layer can be the same or different. The dilated convolution operation can ensure that the resolution remains unchanged while extracting features, that is, it increases the receptive field without losing resolution, thereby avoiding the loss of detailed information as the resolution or feature map size decreases. For example, the receptive field component 424 may include 7 layers of convolutional neural networks for dilated convolution operations, where the dilation coefficients of each layer are 1, 2, 4, 8, 16, 1, and 1 from top to bottom.

[0143] It should be noted that the size of the optical flow map corresponding to the optical flow data obtained by the above-mentioned optical flow estimation model is inconsistent with the first image. Therefore, when used, the optical flow data needs to be upsampled so that the size of the optical flow map corresponding to the upsampled optical flow data is consistent with the size of the first image of the image pair.

[0144] Figure 5A Shown Figures 4A-4D Training flowchart of the optical flow estimation model shown.

[0145] like Figure 5A As shown, the optical flow estimation model 400 can be trained through the following steps S510-S540.

[0146] In step S510 , a plurality of sample image pairs for training are obtained.

[0147] As described above, optical flow estimation model 400 is a self-supervised deep learning-based optical flow estimation model. Therefore, its training samples can be a large amount of unlabeled data. Specifically, only image pairs are required, without the corresponding ground-truth optical flow values. In some embodiments, for road detection problems, the sample image pairs should be road-related images to make the trained model more suitable for optical flow estimation and prediction in road-related images.

[0148] S520: Input the plurality of sample image pairs into an optical flow estimation model to generate optical flow data for each sample image pair.

[0149] During the training of sample images, refer to Figures 4A-4C The optical flow model 400 illustrates the principle and operational process of an original sample image pair. The original sample image pairs are processed by multiple (convolutional) neural network sub-models (e.g., feature extractor 410, optical flow estimator 420 (and their respective components)) to generate their respective optical flow data results. Parameters are then updated based on the results. Samples are then fed back into the model, and the process repeats until the model meets the training requirements. A self-supervisory signal (i.e., a loss function) is used to oversee the training process and parameter adjustments.

[0150] S530 , calculating the consistency loss of each sample image pair using the optical flow data of each sample image pair, wherein the consistency loss includes at least one of a photometric consistency loss and a structural consistency loss between the image pairs.

[0151] In some embodiments, the photometric consistency loss of the first and second images can be used as a first supervisory signal, using the assumption that the luminosity or brightness (e.g., grayscale) of pixels projected from the same object onto different images is consistent (or the assumption of the optical flow method: the luminosity of corresponding pixels in the two images is constant). Furthermore, in some embodiments, because the input sample image pair is an image of the same scene captured with different configurations, the first and second images are similar to a certain extent, and thus the image similarity loss (e.g., in terms of structure, brightness, and contrast) can be used as a second supervisory signal. Optionally, the first and second supervisory signals can be combined to obtain an overall supervisory signal.

[0152] In some embodiments, the first supervisory signal can be represented as a photometric consistency loss between the image pair; the second supervisory signal can be represented as a structural consistency loss between the image pair. Thus, the total supervisory signal can be represented as a consistency loss for the image pair, which includes at least one of photometric consistency and structural consistency. Therefore, the loss function during the training of the optical flow estimation model can be determined based on at least one of the photometric consistency loss and the structural consistency loss for the image pair.

[0153] Photometric consistency can be characterized by the difference between the true pixel value of the first image in the original image pair (first image I1 and second image I2) and the pixel value of the corresponding pixel in the second image calculated based on the optical flow data. Specifically, for a single pixel, the photometric consistency loss can be calculated according to the following formula (3):

[0154] |I1(x1,y1)-I2(x2,y2)| (3)

[0155] Where I1(x1, y1) represents the pixel value of the pixel point (x1, y1) in image I1, which can be a grayscale value or a color value. I2(x2, y2) represents the pixel value at the point (x2, y2) in image I2 calculated using the corresponding optical flow data. If the optical flow data is accurate, the pixel values of the pixels (x1, y1) and (x2, y2) should be close.

[0156] Therefore, the photometric consistency loss of the first image I1 and the second image I2 can be obtained by summing the photometric consistency of all corresponding pixels.

[0157] Structural consistency can be characterized using SSIM (structural similarity index). SSIM is an indicator that measures the similarity between two images. It is calculated as shown in the following formula (4):

[0158] (4)

[0159] Where x represents the first image, y represents the second image, μ x is the mean value of x, μ y is the mean value of y, σ 2 x is the variance of x, σ 2 y is the variance of y, σ xy is the covariance of x and y; c1 = (k1L) 2 , c2=(k2L) 2 It is a constant used to maintain stability, L is the dynamic range of pixel values, k1=0.01, k2=0.03.

[0160] A larger SSIM value indicates a greater structural similarity between the two images. The SSIM value ranges from -1 to 1. When two images are identical, the SSIM value is 1. From the perspective of image composition, SSIM defines structural information as a property that is independent of brightness and contrast and reflects the structure of objects in the scene. It also models distortion as a combination of three different factors: brightness, contrast, and structure. The mean is used as an estimate of brightness, the standard deviation as an estimate of contrast, and the covariance as a measure of structural similarity.

[0161] Therefore, the structural consistency loss of the first image and the second image can be obtained based on formula (4) by using the real pixel value of the pixel point of the first image I1 and the pixel value of the corresponding pixel point in the second image I2 calculated based on the optical flow data. For example, the consistency loss can be 1-SSIM (x, y).

[0162] S540: Adjust the parameters of the optical flow estimation model so that the sum of the consistency losses of the plurality of sample image pairs converges.

[0163] After determining the supervisory signal for each sample image pair, i.e., the consistency loss of the image pair, the consistency of the multiple sample image pairs can be added together to obtain the overall consistency loss. Thus, a loss function can be constructed based on the overall consistency loss to adjust or optimize the model parameters until the model meets the training requirements. Specifically, the loss function can be determined according to the following formulas (5) and (6):

[0164] (5)

[0165] (6)

[0166] Where m represents the number of image pairs, represents the k-th image pair, represents the consistency loss of the k-th image pair, represents the structural consistency loss of the k-th image pair, represents the photometric consistency loss of the k-th image pair, and α is a hyperparameter that can be determined in advance according to specific circumstances.

[0167] In other words, according to formula (6), the consistency loss of a single image pair can be the weighted sum of the photometric consistency loss and the structural consistency loss, and the loss function is the sum of the consistency losses of all image pairs (i.e., the overall consistency loss). In this way, the gradient backpropagation can be performed according to the value of the loss function to update the parameters of the optical flow estimation model until the model meets the training requirements, such as the overall loss is less than the predetermined loss threshold or the overall loss converges.

[0168] Alternatively, since brightness loss is not particularly effective in low-texture areas or areas with repeated features, the estimated depth map can be adjusted in combination with a smoothness prior condition, using edge smoothness loss to ensure the smoothness of the image edges. Therefore, as shown in formulas (7) and (8), a smoothness loss can be added to the loss function.

[0169] (7)

[0170] L s = | (8)

[0171] in, and represent the first-order derivatives in the spatial direction, Represents the depth of the first image in the k-th image pair.

[0172] Need to explain, Figures 4A-4D Each of the components of the optical flow estimation model 400 shown (e.g., the feature extractor 410, the optical flow estimator 420) and the various components included in the optical flow estimator 420 (e.g., the feature transformation component 421, the similarity calculation component 422, the optical flow estimation component 423, and the receptive field expansion component 424) can include one or more layers of neural networks. These neural networks can be trained together during the training process of the optical flow estimation model 400, and the various parameters therein can also be updated as a whole during the training process to achieve the training requirements of the optical flow estimation model 400.

[0173] Figure 5B showing a first image and a second image of an example road taken at different locations, Figure 5C The optical flow estimation model trained according to some embodiments of the present disclosure predicts Figure 5B Optical flow map for the image pair shown.

[0174] like Figure 5B As shown, the first image on the left and the second image on the right are road-related images taken synchronously by cameras placed side by side on the left and right. Figure 5C The pixel points of the optical flow map shown are Figure 5B The pixels of the first image shown correspond to the optical flow. Figure 1 It is usually a color image. For example, the color and brightness of a pixel can be used to represent the optical flow of the pixel. The color represents the displacement direction and the brightness represents the displacement size. Figure 5B and 5C It is easier to see. Figure 5C The optical flow map contour and Figure 5B The original image shown roughly matches.

[0175] Figure 6 Schematically shown Figure 3 The example process of step S330 is shown.

[0176] As described above, the correspondence or matching between pixels in the first and second images can be directly determined based on the optical flow data, which can serve as the basis for subsequent road detection. However, since optical flow data is typically predicted by an optical flow estimation model, the resulting optical flow data may contain inaccurate estimates of pixel optical flow due to errors in the model training process and other factors (such as temporary occlusion of the road in the image). Consequently, the resulting pixel correspondence or matching may not be completely accurate. Therefore, after obtaining pixel correspondence or matching based on the optical flow data, a false match elimination step can be added to obtain a more complete pixel correspondence or matching relationship.

[0177] like Figure 6 As shown, Figure 3 Step S330 shown in FIG. 3 , determining a pixel matching relationship between the first image and the second image based on the optical flow data of the image pair, may include:

[0178] S610, determining a correspondence between pixels in the first image and the second image based on optical flow data of the image pair;

[0179] S620, calculating a matching degree between corresponding pixels in the first image and the second image according to a correspondence relationship between pixels in the first image and the second image;

[0180] S630: Eliminate pixels with a degree of matching lower than a threshold from the pixel correspondence between the first image and the second image to obtain a pixel matching relationship between the first image and the second image.

[0181] In some embodiments, for the sake of clarity, as shown in step S610, the correspondence or matching between the pixel points in the first image and the second image obtained based on the optical flow data can be referred to as the pixel point "correspondence relationship"; and, as shown in step S630, based on the pixel point correspondence relationship obtained in S610, the new correspondence or matching between the pixel points of the first image and the second image obtained through the false matching elimination operation is referred to as the pixel point "matching relationship".

[0182] In some embodiments, the degree of matching of pixels in the first image and the second image can be determined by epipolar geometry. For example, the coordinates of pixels in the first image and the second image obtained at different shooting positions should satisfy the following epipolar constraint formula (9):

[0183] (9)

[0184] Where p1 and p2 are the normalized three-dimensional coordinates of the corresponding pixel points of the first image and the second image on their respective image planes, respectively, and F is the basic matrix (which can be obtained by mathematical methods based on the pixel correspondence and the intrinsic parameter matrix of the camera that captured the first image and the second image, where the camera intrinsic parameter matrix can be calibrated in advance).

[0185] Therefore, according to the principle of epipolar geometry, the matching degree of the pixels in the first image and the second image can be determined according to formula (3). For example, for two corresponding pixels P1 and P2 obtained from the optical flow data, their normalized coordinates are p1 = (x1, y1, 1) and p2 = (x2, y2, 1), respectively. Then the matching degree between the two is value.

[0186] Furthermore, as shown in step S630, to determine whether the matching degree meets the requirements, a matching degree threshold can be pre-set. Subsequently, the matching degree can be compared with the matching degree threshold to determine whether the corresponding pixel correspondence meets the matching requirements. That is, pixels with a value less than the threshold meet the matching requirements and the corresponding pixels are retained; otherwise, the corresponding pixels are eliminated. After this elimination operation, the final matching relationship between the pixels of the first image and the second image can be obtained for use in subsequent operations.

[0187] Figure 7A Shown Figure 3 The example process of step S340 is shown.

[0188] like Figure 7A As shown, Figure 3 The illustrated step S340 - determining the damage condition of the road to be detected based on the pixel matching relationship between the first image and the second image - may include steps S710 - 730 .

[0189] At S710 , the pixel points of the first image and the second image are matched to determine the spatial coordinates of the point cloud of the actual scene corresponding to the first image.

[0190] The point cloud of the actual scene corresponding to the first image is the collection of spatial points in the actual scene corresponding to multiple pixels in the first image. The point cloud may include the spatial points in the actual scene corresponding to some or all pixels in the first image. After obtaining the pixel matching relationship in the image pair, the depth of each location in the actual scene (i.e., the point cloud) corresponding to the pixel points in the first image can be calculated using triangulation methods based on the pixel matching relationship and the pre-calibrated intrinsic parameters of the first and second cameras.

[0191] Figure 7B Figure 2 shows a schematic diagram of the triangulation principle. Figure 7BAs shown in Figure 1, the principle of the triangulation method is: observe the same 3D point P (x, y, z) at different positions O1 and O2 (or photograph point P). Given the 2D projection points P1 and P2 of the 3D point P observed at different positions (or the corresponding pixel points on the phase plane (left) of the first image and the corresponding pixel points on the image plane (right) of the second image obtained by photographing) (i.e., these two pixel points match each other), the depth information of the 3D point P (i.e., the size of O1P and O2P) can be restored using the triangulation relationship.

[0192] Specifically, if Figure 7B As shown in the figure, assume that O1 and O2 are the shooting positions or camera optical centers of the first and second images, respectively. The parallelogram on the left is the image plane of the first image, and the parallelogram on the right is the image plane of the second image. According to the principle of epipolar geometry, let x1 and x2 be the three-dimensional normalized coordinates of pixel point P1 in the first image on the left and pixel point P2 in the second image on the right, respectively. Then they satisfy:

[0193] s1x1 T x1 - s2x1 T Rx2= x1 T t (10)

[0194] s1(Rx2) T x1 - s2(Rx2) T Rx2= (Rx2) T t (11)

[0195] Wherein, R, t are the relative transformation coefficients of the camera coordinate systems corresponding to the first image and the second image, s1 and s2 are the depths of point P relative to the first image pixel point P1 and the second image pixel point P2, respectively.

[0196] According to the principle of epipolar geometry, Figure 6 From the relevant description, we can know that the basic matrix F can be obtained by mathematical methods based on the pixel correspondence and the camera's intrinsic parameter matrix, and R and t can be decomposed through the basic matrix F. Therefore, a quadratic linear equation system can be obtained by combining formulas (10) and (11), and Cramer's rule can be used to solve s1.

[0197] In some embodiments, based on the principle of epipolar geometry, the depths s1 and s2 can also be solved by the following formula (12):

[0198] s1x1^x1= 0 =s2 x1^ R x2 + x1^ t (12)

[0199] Where x1^ represents the antisymmetric matrix of x1.

[0200] In some embodiments, as Figure 7B As shown in Figure 1, due to the presence of noise, the line connecting O1P1 and O2P2 may not intersect at point P, that is, the formulas (10)-(12) obtained from the epipolar geometry principle cannot be accurately established. Therefore, the least squares method can be used to find an approximate solution for the depth s1. Alternatively, other methods can be used to calculate the pixel depth information.

[0201] After the depth is determined, the three-dimensional coordinates of the point cloud in the actual scene can be obtained using a geometric method based on the depth of the point cloud and the two-dimensional coordinates of each pixel point in the first image.

[0202] At S720 , a surface equation of the road to be detected is fitted according to the spatial coordinates of the point cloud of the actual scene.

[0203] After obtaining the coordinates of the three-dimensional point cloud in the actual scene of the road to be detected, the surface equation of the road to be detected can be fitted using the point cloud coordinates according to the surface fitting algorithm.

[0204] For example, we can first assume that the equation of the road surface is:

[0205] + (13)

[0206] Subsequently, the parameters a1,…,a6 in the surface equation (13) are fitted using least squares according to the coordinates of some or all (the number of points is greater than 6) points in the point cloud, thereby obtaining the surface equation of the road to be detected.

[0207] Optionally, other fitting algorithms may also be used to fit the quadratic surface satisfied by the road to be detected.

[0208] At S730 , the damage condition of the road to be detected is determined according to the surface equation of the road to be detected and the coordinates of the point cloud of the actual scene.

[0209] After obtaining the surface equation of the road to be tested, it is obvious that the coordinates of a certain point in the actual scene can be used to determine the surface equation and obtain the position of the point relative to the surface equation. Therefore, the presence of road damage can be determined based on the position of the point. For example, by substituting the coordinates of the point into the surface equation, it is calculated that multiple points located below the surface appear in a certain area. For example, if the number of points exceeds a predetermined number threshold and the distances from these points to the surface exceed a predetermined distance threshold, then the area may be a damaged area, especially a pothole area.

[0210] In some embodiments of the present disclosure, road damage can include at least one of potholes and cracks. Potholes are depressions relative to the road surface caused by road damage, while cracks are linear or curved gaps caused by aging or road quality issues. Potholes can be described using the following parameters: number, center point location, area, and depth (maximum depth); cracks can be characterized using the following parameters: number, length, depth, width, and orientation of cracks. Clearly, all of these parameters (including pothole and crack characteristics) can be obtained using geometric methods using point cloud coordinates and surface equations.

[0211] Figure 7C FIG. 1 shows a schematic diagram of the visualization results of the road detection method according to some embodiments of the present disclosure. Figure 7C As shown, the specific location of potholes in the road image is detected by the optical flow method based on the road image to be detected disclosed in the present invention ( Figure 7C As shown in the box). Experiments and statistics show that the road detection method according to some embodiments of the present disclosure is effective, with a recall rate of pothole detection greater than 90% and an accuracy or precision greater than 70%, which are much higher than the industry average.

[0212] Figure 8 The following schematically illustrates an example block diagram of a road detection device 800 according to some embodiments of the present disclosure. The road detection device 800 may include an acquisition module 810 , a first determination module 820 , a second determination module 830 , and a third determination module 840 .

[0213] The acquisition module 810 can be configured to acquire an image pair for the road to be inspected, the image pair comprising a first image and a second image captured at different shooting positions. The first determination module 820 can be configured to input the image pair into an optical flow estimation model to generate optical flow data for the image pair. The second determination module 830 can be configured to determine the pixel matching relationship between the first image and the second image based on the optical flow data of the image pair. The third determination module 840 can be configured to determine the damage status of the road to be inspected based on the pixel matching relationship between the first image and the second image.

[0214] Therefore, in the road detection device based on optical flow technology according to some embodiments of the present disclosure, on the one hand, the detection accuracy is improved because it is not strictly limited to the use of binocular cameras during image acquisition; on the other hand, by using optical flow data to determine the pixel matching relationship, the defects of the binocular camera-based algorithm are avoided, thereby significantly simplifying the calculation process and reducing the detection and camera maintenance costs.

[0215] It should be noted that the various modules described above can be implemented in software or hardware or a combination of both. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0216] Figure 9 Schematically illustrates an example block diagram of a computing device 900 according to some embodiments of the present disclosure. The computing device 900 may represent a device for implementing the various means or modules described herein and / or performing the various methods described herein. The computing device 900 may be, for example, a server, a desktop computer, a laptop computer, a tablet, a smart phone, a smart watch, a wearable device, or any other suitable computing device or computing system, which may include various levels of devices ranging from full-resource devices with a large amount of storage and processing resources to low-resource devices with limited storage and / or processing resources. In some embodiments, the above description of the computing device 900 may be a computer system or a computing system. Figure 8 The described road detection device 800 may be implemented in one or more computing devices 900 .

[0217] like Figure 9 As shown, the example computing device 900 includes a processing system 901, one or more computer-readable media 902, and one or more I / O interfaces 903 that are communicatively coupled to each other. Although not shown, the computing device 900 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures. Alternatively, it may also include control and data lines.

[0218] Processing system 901 represents functionality that uses hardware to perform one or more operations. Thus, processing system 901 is illustrated as including hardware elements 904 that can be configured as processors, functional blocks, and the like. This can include hardware implementation as application-specific integrated circuits (ASICs) or other logic devices formed using one or more semiconductors. Hardware elements 904 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be comprised of semiconductor(s) and / or transistors (e.g., an electronic integrated circuit (IC)). In such a context, processor-executable instructions may be electronically executable instructions.

[0219] Computer-readable media 902 is illustrated as including memory / storage 905. Memory / storage 905 represents memory / storage associated with one or more computer-readable media. Memory / storage 905 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 905 may include fixed media (e.g., RAM, ROM, fixed hard drives, etc.) as well as removable media (e.g., flash memory, removable hard drives, optical disks, etc.). For example, memory / storage 905 may be used to store the first audio of the first category of users, the queued list of requests, and the like, as mentioned in the above embodiments. Computer-readable media 902 may be configured in various other ways, as further described below.

[0220] One or more I / O (input / output) interfaces 903 represent functionality that allows a user to enter commands and information into the computing device 900 and also allows information to be displayed to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., capable of detecting gestures without touch using visible or invisible wavelengths (such as infrared frequencies)), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like. For example, in the embodiments described above, the first category of users and the second category of users can use the input interfaces on their respective terminal devices to initiate requests and record audio and / or video, and can use the output interfaces to view various notifications, watch videos, or listen to audio, and the like.

[0221] The computing device 900 also includes a road detection strategy 906. The road detection strategy 906 can be stored as a computer program instruction in the memory / storage device 905, or can be hardware or firmware. The road detection strategy 906 can be implemented together with the processing system 901 and the like. Figure 8 The entire functions of each module of the road detection device 800 are described.

[0222] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," etc. generally refer to software, firmware, hardware, or a combination thereof. A feature of the techniques described herein is that they are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0223] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by the computing device 900. By way of example and not limitation, computer-readable media may include "computer-readable storage media" and "computer-readable signal media."

[0224] "Computer-readable storage media" refers to media and / or devices capable of persistently storing information, as opposed to mere signal transmissions, carrier waves, or signals themselves. Thus, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media includes hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information, such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, hard drives, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture suitable for storing the desired information and accessed by a computer.

[0225] "Computer-readable signal media" refers to signal-bearing media configured to transmit instructions to the hardware of computing device 900, such as via a network. Signal media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also includes any information transmission media. By way of example, and not limitation, signal media include wired media such as a wired network or direct connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0226] As previously described, hardware elements 904 and computer-readable media 902 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware that, in some embodiments, can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be considered processing devices that execute program tasks defined by the instructions, modules, and / or logic embodied by the hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0227] The aforementioned combinations may also be used to implement the various techniques and modules described herein. Thus, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 904. The computing device 900 may be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Thus, for example, by using a computer-readable storage medium and / or hardware elements 904 of a processing system, a module may be implemented as a module executable by the computing device 900 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 900 and / or processing systems 901 to implement the techniques, modules, and examples described herein.

[0228] The techniques described herein may be supported by these various configurations of computing device 900 and are not limited to the specific examples of the techniques described herein.

[0229] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer programs. For example, embodiments of the present disclosure provide a computer program product comprising a computer program carried on a computer-readable medium, the computer program including program code for executing at least one step of the method embodiments of the present disclosure.

[0230] In some embodiments of the present disclosure, one or more computer-readable storage media are provided, on which computer-readable instructions are stored. When executed, these computer-readable instructions implement the road detection method according to some embodiments of the present disclosure. The various steps of the road detection method according to some embodiments of the present disclosure can be converted into computer-readable instructions through programming and then stored in a computer-readable storage medium. When such a computer-readable storage medium is read or accessed by a computing device or computer, the computer-readable instructions therein are executed by a processor on the computing device or computer to implement the road detection method according to some embodiments of the present disclosure.

[0231] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0232] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment, or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be performed in a sequence other than as shown or discussed (including in a substantially simultaneous manner or in reverse order depending on the functions involved), which should be understood by those skilled in the art to which the embodiments of the present disclosure pertain.

[0233] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0234] It should be understood that various parts of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0235] Those skilled in the art will appreciate that all or part of the steps of the method of the above embodiment may be accomplished through hardware associated with program instructions, and the program may be stored in a computer-readable storage medium, which, when executed, includes executing one or a combination of the steps of the method embodiment.

[0236] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

Claims

1. A road detection method, comprising: Acquire an image pair for the road to be inspected, the image pair comprising a first image and a second image acquired synchronously at different shooting positions using two monocular cameras; determining optical flow data for the image pair using a self-supervised optical flow estimation model, the optical flow data indicating positional offsets of pixels of the first image relative to the second image; determining a correspondence between pixels in the first image and the second image based on the optical flow data of the image pair; Calculating the matching degree between corresponding pixels using the principle of epipolar geometry based on the correspondence between pixels in the first image and the second image and the intrinsic parameter matrix of the monocular camera; determining a pixel matching relationship between the first image and the second image based on a pixel correspondence relationship and a matching degree between corresponding pixels in the first image and the second image; Determine the damage of the road to be detected based on the pixel matching relationship between the first image and the second image. The self-supervised optical flow estimation model is trained by the following steps: Acquire a plurality of sample image pairs for training, each sample image pair comprising a first sample image and a second sample image of the same scene that are synchronously acquired at different shooting positions; Inputting the plurality of sample image pairs into a self-supervised optical flow estimation model to generate optical flow data for each sample image pair; calculating the consistency loss of the sample image pair using the optical flow data of each sample image pair, wherein the consistency loss of the sample image pair includes a weighted sum of a photometric consistency loss and a structural consistency loss between the sample image pairs; Parameters of the optical flow estimation model are adjusted so that the sum of the consistency losses of the plurality of sample image pairs converges.

2. The method according to claim 1, wherein The determining the damage condition of the road to be inspected based on the pixel matching relationship between the first image and the second image includes: Determining the spatial coordinates of a point cloud of an actual scene corresponding to the first image based on a pixel matching relationship between the first image and the second image; Fitting a surface equation of the road to be detected according to the spatial coordinates of the point cloud of the actual scene; The damage condition of the road to be detected is determined according to the surface equation of the road to be detected and the spatial coordinates of the point cloud of the actual scene.

3. The method according to claim 2, wherein: The determining of the damage condition of the road to be detected based on the surface equation of the road to be detected and the spatial coordinates of the point cloud of the actual scene includes: determining at least one of the number, location, area, and depth of potholes on the road to be detected based on the surface equation of the road to be detected and the spatial coordinates of the point cloud of the actual scene; At least one of the number, position, width, length, and orientation of cracks on the road to be detected is determined based on the surface equation of the road to be detected and the spatial coordinates of the point cloud of the actual scene.

4. The method of claim 1 , wherein determining optical flow data for the image pair using a self-supervised optical flow estimation model comprises: Using a feature extractor of a self-supervised optical flow estimation model, constructing a feature pyramid comprising n layers of feature maps according to the image pair, where n>=2, wherein the first layer of the feature pyramid is extracted from the image pair, the kth layer of the feature map is extracted from the k-1th layer of the feature map, and k is a positive integer greater than 1 and less than or equal to n, and An optical flow estimator of a self-supervised optical flow estimation model is used to generate optical flow data of the image pair according to a feature pyramid.

5. The method according to claim 4, wherein the generating the optical flow data of the image pair according to the feature pyramid using the optical flow estimator of the self-supervised optical flow estimation model comprises: Generate optical flow data of the n-th layer feature map according to the n-th layer feature map of the feature pyramid and the preset initial optical flow data; The optical flow data of the i-th layer feature map of the feature pyramid is calculated in order from bottom to top through the following steps, where i is a positive integer greater than or equal to 1 and less than or equal to n-1: the optical flow data of the i-th layer feature map is generated based on the optical flow data of the i+1-th layer feature map and the i-th layer feature map; as well as Based on the optical flow data of the feature map of the first layer of the feature pyramid, the optical flow data of the image pair is generated.

6. The method according to claim 5, wherein: Each layer of the feature pyramid includes a first feature map and a second feature map corresponding to the first image and the second image, respectively, and The step of generating the optical flow data of the i-th layer feature map based on the optical flow data of the i+1-th layer feature map and the i-th layer feature map includes: performing a view transformation on the second feature map in the i-th layer feature map according to the optical flow data of the i+1-th layer feature map to obtain a transformed second feature map, wherein the view transformation includes upsampling the optical flow data of the i+1-th layer feature map, and performing a first transformation on the second feature map of the i-th layer feature map using the optical flow data of the upsampled i+1-th layer feature map; Determine the similarity between a first feature map and a transformed second feature map of the i-th layer feature map; Based on the first feature map of the i-th layer feature map, the similarity between the first feature map of the i-th layer feature map and the transformed second feature map, and the optical flow data of the i+1-th layer feature map, the optical flow data of the i-th layer feature map is generated.

7. The method according to claim 6, wherein generating the optical flow data of the image pair based on the optical flow data of the feature map of the first layer of the feature pyramid comprises: The receptive field of the optical flow map corresponding to the optical flow data of the first layer feature map is expanded to generate optical flow data of the image pair, where the receptive field of the optical flow map represents the size of an area mapped by pixels in the optical flow map on the first image in the image pair.

8. The method according to claim 6, wherein the view transformation further comprises: A mask operation is performed on the second feature map after the first transformation, wherein the mask operation is used to estimate an occlusion condition of a pixel point in the second feature map after the first transformation.

9. The method according to claim 6, wherein: Determining the similarity between the first feature map of the i-th layer feature map and the transformed second feature map includes: increasing the length and width of the transformed second feature map to obtain an increased second feature map; The first feature map of the i-th layer feature map is used as a convolution kernel, and a convolution operation is performed on the enlarged second feature map to obtain a similarity between the first feature map of the i-th layer feature map and the transformed second feature map.

10. The method of claim 1, wherein the consistency loss further comprises a smoothness loss of the sample image pair.

11. A road detection device comprising: An acquisition module configured to acquire an image pair for a road to be inspected, the image pair comprising a first image and a second image synchronously acquired at different shooting positions using two monocular cameras; A first determination module is configured to determine optical flow data of the image pair using a self-supervised optical flow estimation model, wherein the optical flow data indicates a position offset of a pixel point of the first image relative to the second image, wherein the self-supervised optical flow estimation model is trained by the following steps: obtaining a plurality of sample image pairs for training, each sample image pair comprising a first sample image and a second sample image of the same scene synchronously acquired at different shooting positions; inputting the plurality of sample image pairs into the self-supervised optical flow estimation model to generate optical flow data for each sample image pair; calculating a consistency loss of the sample image pair using the optical flow data of each sample image pair, wherein the consistency loss of the sample image pair comprises a weighted sum of a photometric consistency loss and a structural consistency loss between the sample image pairs; and adjusting parameters of the optical flow estimation model so that the sum of the consistency losses of the plurality of sample image pairs converges; a second determination module configured to determine a correspondence between pixels in the first image and the second image based on the optical flow data of the image pair, calculate a matching degree between corresponding pixels based on the correspondence between the pixels in the first image and the second image and an intrinsic parameter matrix of the monocular camera using the principle of epipolar geometry, and determine a pixel matching relationship between the first image and the second image based on the correspondence between the pixels in the first image and the second image and the matching degree between the corresponding pixels; The third determining module is configured to determine the damage condition of the road to be detected based on the pixel matching relationship between the first image and the second image.

12. A computing device comprising: memory and processor, A computer program is stored in the memory, and when the computer program is executed by the processor, it causes the processor to perform the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions implement the method according to any one of claims 1 to 10 when executed.

Images