Obstacle identification method and device, electronic device, and storage medium

Abstract

Embodiments of the present application provide a kind of obstacle identification method and device, electronic equipment and storage medium, belong to image processing technical field.The method comprises: obtaining the infrared image of target object;Edge repair is carried out to infrared image, and edge recovery image is obtained;According to infrared image, light flow field construction is carried out, and temperature conservation light flow field is obtained;According to edge recovery image, confidence is extracted, and obstacle confidence chart is obtained;According to obstacle confidence chart, edge recovery image and temperature conservation light flow field, obstacle identification is carried out.The accuracy of the present application embodiment can be improved for obstacle identification.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an obstacle recognition method and apparatus, electronic device and storage medium. Background Technology

[0002] Obstacle recognition refers to the detection and identification of target objects in images or sensor data to determine obstacles. For example, during vehicle operation, an onboard infrared imaging device acquires infrared images of the environment ahead, and the target areas in the image are analyzed to identify vehicles, pedestrians, or other potential obstacles on the road. However, during obstacle recognition, the edges of the target object are easily blurred or diffused due to the complex thermal background of the infrared image, making it difficult to accurately extract the obstacle area and thus affecting the accuracy of obstacle recognition. Therefore, improving the accuracy of obstacle recognition has become an urgent problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide an obstacle recognition method, device, electronic device, and storage medium, which aims to improve the accuracy of obstacle recognition.

[0004] To achieve the above objectives, a first aspect of this application proposes an obstacle recognition method, the method comprising: Acquire infrared images of the target object; The infrared image is subjected to edge restoration to obtain an edge-restored image; An optical flow field is constructed based on the infrared image to obtain a temperature-conserving optical flow field. Confidence is extracted from the edge reconstruction image to obtain an obstacle confidence map; Obstacle identification is performed based on the obstacle confidence map, the edge restoration image, and the temperature-conserving optical flow field.

[0005] In some embodiments, the edge restoration of the infrared image to obtain an edge-restored image includes: The infrared image is subjected to thermal target region detection to obtain a thermal target image; Based on the thermal target image, an adaptive point diffusion model is constructed to obtain the model. The infrared image is edge-restored by constructing the adaptive point diffusion model to obtain the edge-restored image.

[0006] In some embodiments, the step of extracting confidence scores from the edge reconstruction image to obtain an obstacle confidence map includes: The edge-restored image is subjected to superpixel segmentation to obtain superpixel sub-images; Temperature modeling is performed on the superpixel sub-image to obtain a mixture of Gaussian thermal distribution models; Based on the hybrid Gaussian heat distribution model and preset environmental thermal inertia parameters, a dynamic segmentation threshold is generated to obtain the confidence threshold. The confidence level of the superpixel sub-image is calculated based on the confidence threshold to obtain the obstacle confidence map.

[0007] In some embodiments, the step of constructing an optical flow field based on the infrared image to obtain a temperature-conserving optical flow field includes: The infrared image is subjected to vector calculation to obtain the optical flow vector; Temperature data is obtained by performing temperature calculations on the infrared image. The infrared image is subjected to grayscale calculation to obtain the image grayscale. The temperature-conserving optical flow field is obtained by calculating based on the optical flow vector, the temperature data, and the image grayscale.

[0008] In some embodiments, the step of constructing an optical flow field based on the infrared image to obtain a temperature-conserving optical flow field further includes: Obtain motion parameters; The nominal optical flow field is obtained by calculating based on the aforementioned motion parameters; The optical flow difference is obtained by performing pixel-by-pixel residual calculation based on the nominal optical flow field and the temperature-conserved optical flow field. If the optical flow difference is greater than a preset threshold, the temperature-conserving optical flow field is suppressed.

[0009] Temperature is extracted from the thermal target image to obtain the target temperature difference; The distance to the target is obtained by performing distance calculation on the thermal target image; Based on the target temperature difference and the target distance, a model is constructed to obtain the adaptive point diffusion model, which is shown in the following equation: , The adaptive construction point diffusion model is as follows: , The target temperature difference, The target distance is... Atmospheric permeability, For the system parameters of the infrared camera, The coefficient of thermal stability. It is a basic diffusion term.

[0010] The mean value was obtained by performing mean statistics on the mixed Gaussian heat distribution model. Variance statistics were performed on the mixture Gaussian heat distribution model to obtain the variance values; The confidence threshold is calculated based on the environmental thermal inertia parameters, the mean value, and the variance value, as shown in the following formula: , in, The confidence threshold is... For the first Each superpixel block at time The mean value, For the first The variance value of each superpixel block at time t. and For adjustment coefficients, To represent the time variable during the integration process, In order to be in Temperature difference over time This represents the time constant of environmental thermal inertia.

[0011] To achieve the above objectives, a second aspect of this application provides an obstacle recognition device, the device comprising: The data acquisition module is used to acquire infrared images of the target object; The physical correction module is used to perform edge repair on the infrared image to obtain an edge-restored image; The mechanism-driven module is used to construct an optical flow field based on the infrared image to obtain a temperature-conserving optical flow field. The statistical inference module is used to extract confidence based on the edge restoration image to obtain an obstacle confidence map. The inference output module is used to identify obstacles based on the obstacle confidence map, the edge restoration image, and the temperature-conserving optical flow field, and to obtain the obstacle category and obstacle location.

[0012] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0014] This application proposes an obstacle recognition method, apparatus, electronic device, and storage medium. It acquires an infrared image of a target object and performs edge restoration processing on the infrared image to obtain an edge-restored image. This reduces the blurring or diffusion of target edges caused by thermal halo effects and complex thermal backgrounds in the infrared image, making the target outline clearer and facilitating accurate extraction of subsequent obstacle regions. Simultaneously, an optical flow field is constructed based on the infrared image to obtain a temperature-conserving optical flow field. This utilizes the stable characteristics of temperature attributes in the infrared image to constrain pixel motion, improving the reliability of motion information. Further, confidence is extracted from the edge-restored image to obtain an obstacle confidence map, thereby improving the accuracy of potential obstacle region identification. Based on this, the obstacle confidence map, edge-restored image, and temperature-conserving optical flow field are fused for obstacle recognition, comprehensively utilizing image structure information, motion information, and region confidence information to improve the accuracy of obstacle recognition. Attached Figure Description

[0015] Figure 1 This is a flowchart of the obstacle recognition method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3 yes Figure 2 The flowchart of step S202 in the document; Figure 4 yes Figure 1 The flowchart of step S103 in the process; Figure 5 This is a flowchart of an obstacle recognition method provided in another embodiment of this application; Figure 6 yes Figure 1 The flowchart of step S104 in the process; Figure 7 yes Figure 6 The flowchart of step S603 in the process; Figure 8 This is a connection diagram of an obstacle recognition module provided in an embodiment of this application; Figure 9 This is a schematic diagram of the obstacle recognition device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] Obstacle recognition refers to the detection and identification of target objects in images or sensor data to determine obstacles. For example, during vehicle operation, an onboard infrared imaging device acquires infrared images of the environment ahead, and the target areas in the image are analyzed to identify vehicles, pedestrians, or other potential obstacles on the road. However, during obstacle recognition, the edges of the target object are easily blurred or diffused due to the complex thermal background of the infrared image, making it difficult to accurately extract the obstacle area and thus affecting the accuracy of obstacle recognition. Therefore, improving the accuracy of obstacle recognition has become an urgent problem to be solved.

[0020] Based on this, embodiments of this application provide an obstacle recognition method and apparatus, an electronic device and a storage medium, aiming to improve the accuracy of obstacle recognition.

[0021] This application provides an obstacle recognition method, apparatus, electronic device, and storage medium, which are specifically described through the following embodiments. First, the obstacle recognition method in this application is described.

[0022] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0023] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0024] The obstacle recognition method provided in this application relates to the field of image processing technology. The obstacle recognition method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the obstacle recognition method, but is not limited to the above forms.

[0025] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0026] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0027] Figure 1 This is an optional flowchart of the obstacle recognition method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0028] Step S101: Acquire an infrared image of the target object; Step S102: Perform edge restoration on the infrared image to obtain an edge-restored image; Step S103: Construct an optical flow field based on the infrared image to obtain a temperature-conserving optical flow field; Step S104: Confidence is extracted from the edge restoration image to obtain the obstacle confidence map; Step S105: Obstacle identification is performed based on the obstacle confidence map, edge restoration image, and temperature-conserving optical flow field.

[0029] Steps S101 to S105 of this embodiment involve acquiring an infrared image of the target object and performing edge restoration processing on the infrared image to obtain an edge-restored image. This reduces the blurring or diffusion of the target edge caused by thermal halo effect and complex thermal background in the infrared image, making the target outline clearer and facilitating accurate extraction of subsequent obstacle regions. Simultaneously, an optical flow field is constructed based on the infrared image to obtain a temperature-conserving optical flow field. This utilizes the stable characteristics of temperature attributes in the infrared image to constrain pixel motion, improving the reliability of motion information. Further, confidence is extracted from the edge-restored image to obtain an obstacle confidence map, thereby improving the accuracy of identifying potential obstacle regions. Based on this, the obstacle confidence map, the edge-restored image, and the temperature-conserving optical flow field are fused for obstacle recognition, thus comprehensively utilizing image structure information, motion information, and region confidence information to improve the accuracy of obstacle recognition.

[0030] In step S101 of some embodiments, the target object refers to an object within the monitoring area that may affect the operation of the carrier. For example, in a car driving scenario, the target object can be a vehicle, pedestrian, bicycle, animal, or road obstacle. When the vehicle monitors the environment ahead using an onboard infrared imaging device, the onboard infrared imaging device can perform infrared imaging on the aforementioned target object, thereby acquiring an infrared image of the target object. An infrared image is an image formed by receiving the infrared radiation signal radiated or reflected by the target object through an infrared imaging device and converting the infrared radiation signal into image information. Unlike visible light images, infrared images mainly reflect the temperature distribution or thermal radiation characteristics of the target object, thus enabling effective imaging of the target object even at night, in low light, or in complex environmental conditions.

[0031] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S203: Step S201: Detect the thermal target region in the infrared image to obtain a thermal target image; Step S202: Construct a model based on the thermal target image to obtain an adaptive point diffusion model; Step S203: Perform edge repair on the infrared image based on the adaptively constructed point diffusion model to obtain the edge-restored image.

[0032] Steps S201 to S203, as illustrated in this embodiment, involve detecting thermal target regions in an infrared image to obtain a thermal target image, thereby enabling the location of target regions with obvious thermal features within the infrared image. Based on this, a model is constructed using the thermal target image to obtain an adaptively constructed point diffusion model. This model can adaptively adjust according to the characteristics of the thermal target region, thus more accurately representing the diffusion effect generated during infrared imaging. Furthermore, edge restoration processing is performed on the infrared image based on the adaptively constructed point diffusion model to obtain an edge-restored image. This reduces the blurring or diffusion of target edges caused by the diffusion effect during infrared imaging, making the target outline clearer and facilitating the accurate extraction of subsequent obstacle regions, thereby improving the accuracy of obstacle recognition.

[0033] In step S201 of some embodiments, the thermal target image refers to an image obtained by extracting target areas with obvious thermal radiation characteristics from an infrared image. Since infrared images can reflect the temperature distribution or thermal radiation intensity of various areas in a scene, areas with temperatures significantly higher or lower than the surrounding background typically exhibit prominent thermal target areas. By detecting thermal target areas in an infrared image, target areas with significant temperature differences can be separated from the background area, thereby forming a thermal target image.

[0034] In some implementations, temperature difference analysis can be performed on the image based on the temperature information corresponding to the pixel grayscale values ​​in the infrared image to identify regions with temperatures higher or lower than a preset threshold as candidate thermal target regions. Subsequently, by performing connectivity analysis or region filtering on the candidate thermal target regions, regions with small areas or those that do not meet the preset conditions are removed, thereby retaining regions that may correspond to the target object and generating a thermal target image.

[0035] Please see Figure 3 In some embodiments, step S202 may include, but is not limited to, steps S301 to S303: Step S301: Extract the temperature from the thermal target image to obtain the target temperature difference; Step S302: Calculate the distance to the thermal target image to obtain the target distance; Step S303: Based on the target temperature difference and target distance, construct a model to obtain an adaptive construction point diffusion model.

[0036] Steps S301 to S303, as illustrated in this embodiment, involve extracting the temperature of the thermal target image to obtain the target temperature difference, thereby acquiring the thermal radiation characteristic information of the target area. Simultaneously, distance calculation is performed on the thermal target image to obtain the target distance, thus acquiring the spatial position information between the target and the imaging device. Based on this, a model is constructed according to the target temperature difference and target distance to obtain an adaptively constructed point diffusion model. This model can dynamically adjust based on the target's temperature characteristics and spatial distance information, thereby more accurately representing the diffusion effect caused by thermal radiation propagation and imaging system factors during infrared imaging. This improves the model's fit to the actual imaging process, facilitating more accurate edge repair processing of the infrared image, improving the target contour restoration effect, and further enhancing the accuracy of obstacle recognition.

[0037] In steps S301 to S303 of some embodiments, temperature extraction can be achieved by mapping the correspondence between pixel grayscale values ​​and temperature using an infrared imaging device. Specifically, when acquiring an infrared image, the infrared imaging device can establish a mapping relationship between pixel grayscale values ​​and target surface temperature based on the device's radiometric calibration parameters, thereby converting the grayscale values ​​of corresponding pixels in the thermal target image into temperature values.

[0038] Distance calculation can be performed by combining the imaging features of the target in the image with the parameters of the imaging device. For example, given the focal length and pixel size parameters of the infrared imaging device, the distance between the target and the imaging device can be estimated based on the size information of the thermal target region in the image.

[0039] The adaptive construction method of the point diffusion model is shown in equation (1): (1), Among them, its scale parameter σ is inversely proportional to the temperature difference ΔT, and also to the target distance d and the atmospheric transmittance parameter. This is correlated to simulate the diffusion effect of thermal radiation during its propagation in space, and can be obtained by looking up a table. The target temperature in the thermal target image is... Non-target temperature is , Target temperature Non-target temperature The absolute value of the difference between them; The target distance; These are the system parameters of the infrared camera, used to characterize the influence of the optical system and detector characteristics of the infrared imaging device on the point spread effect; Basic diffusion term, The coefficient of thermal stability. These are atmospheric transmittance parameters, all of which are preset values.

[0040] In step S203 of some embodiments, the edge restoration image refers to the image obtained after repairing the edge blurring phenomenon in the infrared image caused by the diffusion effect of the imaging system or the propagation of thermal radiation. Due to the point diffusion effect during infrared imaging, the true edges of the target object are easily diffused or blurred during imaging, resulting in an unclear target outline. By performing edge repair processing on the infrared image, the true boundary information of the target object can be restored, thereby obtaining an edge restoration image with clearer edges.

[0041] In some implementations, infrared images can be deconvolved based on the adaptively constructed point spread model to achieve image edge restoration. Specifically, a corresponding point spread function can first be constructed according to the point spread model, and the point spread function can be transformed to the frequency domain; then, a Fourier transform can be performed on the infrared image to obtain the image spectrum, and deconvolution operations can be performed in the frequency domain in conjunction with the point spread function to reduce the blurring effect caused by diffusion; finally, the restored image can be obtained by performing an inverse Fourier transform on the processed frequency domain data, thereby obtaining the edge-restored image. Through the above method, the edge diffusion phenomenon generated during infrared imaging can be reduced, making the outline of the target object clearer.

[0042] Please see Figure 4 In some embodiments, step S103 may include, but is not limited to, steps S401 to S404: Step S401: Perform vector calculation on the infrared image to obtain the optical flow vector; Step S402: Calculate the temperature of the infrared image to obtain temperature data; Step S403: Perform grayscale calculation on the infrared image to obtain the image grayscale; Step S404: Calculate the temperature-conserving optical flow field based on the optical flow vector, temperature data, and image grayscale.

[0043] Steps S401 to S404, as illustrated in this embodiment, involve performing vector calculations on the infrared image to obtain an optical flow vector, thereby acquiring motion information of pixels in the image over time. Simultaneously, temperature calculations are performed on the infrared image to obtain temperature data, thereby acquiring thermal radiation characteristic information corresponding to each pixel in the image. Furthermore, grayscale calculations are performed on the infrared image to obtain image grayscale, thereby acquiring brightness change information of the image. Based on this, a temperature-conserving optical flow field is obtained through joint calculations using the optical flow vector, temperature data, and image grayscale. This allows the optical flow field to reflect pixel motion characteristics while being constrained by the temperature attribute information of the infrared image, thereby improving the stability and accuracy of optical flow estimation. This facilitates the acquisition of more reliable motion information and further enhances the accuracy of obstacle recognition.

[0044] In some embodiments, the calculation method of the temperature-conserving optical flow field in steps S401 to S404 is as shown in equation (2): (2), where I is the image gray level, u and v are the optical flow vectors to be solved, α is the weight of the smoothing term, β is the weight of the temperature conservation term, and T is the physical temperature mapping value of the pixel; the third term of the energy functional is the temperature conservation constraint term, which is used to penalize vectors with inconsistent temperature attributes before and after motion. , These represent the x and y coordinates of a pixel in the image, respectively.

[0045] Please see Figure 5 In some embodiments, after step S404, the obstacle recognition method further includes, but is not limited to, steps S501 to S504: Step S501: Obtain motion parameters; Step S502: Calculate the nominal optical flow field based on the motion parameters; Step S503: Calculate the pixel-by-pixel residual based on the nominal optical flow field and the temperature-conserved optical flow field to obtain the optical flow difference value; Step S504: If the optical flow difference is greater than a preset threshold, suppress the temperature-conserving optical flow field.

[0046] Steps S501 to S504, as illustrated in this embodiment, obtain motion parameters and calculate a nominal optical flow field based on these parameters. This allows for the prediction of the theoretical motion of pixels in the image based on the carrier's own motion state. Furthermore, a pixel-by-pixel residual calculation is performed between the nominal optical flow field and the temperature-conserving optical flow field to obtain an optical flow difference value. This value is used to evaluate the difference between the actual estimated optical flow information and the theoretical motion information. When the optical flow difference value exceeds a preset threshold, the temperature-conserving optical flow field is suppressed. This process identifies and weakens unreliable optical flow information caused by noise interference, background changes, or abnormal motion, improving the reliability and stability of the optical flow field. This facilitates obtaining more accurate motion information and provides a more reliable data foundation for subsequent obstacle recognition, thereby enhancing the accuracy of obstacle recognition.

[0047] In step S501 of some embodiments, the motion parameters refer to parameters used to characterize the motion state information of the carrier during motion. For example, in a car driving scenario, the motion parameters may include vehicle speed information, acceleration information, angular velocity information, attitude information, or displacement information, etc. These motion parameters can be acquired by an inertial measurement unit, vehicle speed sensor, or other motion detection device in the vehicle system.

[0048] In step S502 of some embodiments, the nominal optical flow field refers to the theoretical motion vector field of pixels in the image predicted based on the motion state of the carrier itself, used to describe the expected motion of each pixel in the image when only the carrier motion is considered. In other words, the nominal optical flow field reflects the overall motion trend of the image caused by the carrier motion under ideal conditions.

[0049] In one embodiment, given the known velocity information, angular velocity information, or attitude change information of the carrier, the overall displacement change between adjacent image frames can be estimated based on the motion parameters, and this displacement change can be mapped to the image pixel coordinate system to obtain the theoretical motion vector of each pixel in the image, thereby forming a nominal optical flow field.

[0050] In step S503 of some embodiments, pixel-by-pixel residual calculation refers to calculating the difference between the optical flow vectors of the nominal optical flow field and the temperature-conserving optical flow field at the corresponding pixel positions in the image pixel coordinate system. The optical flow vector at each pixel position in the nominal optical flow field is compared with the optical flow vector at the corresponding pixel position in the temperature-conserving optical flow field, and the optical flow difference at that pixel position is obtained by calculating the difference between the two optical flow vectors.

[0051] In step S504 of some embodiments, when the optical flow difference at a certain pixel location is greater than the preset threshold, it can be considered that there is a large deviation between the temperature-conserving optical flow vector at that location and the carrier motion prediction result, thus potentially belonging to unreliable optical flow information. In this case, the temperature-conserving optical flow vector at the corresponding pixel location can be suppressed, for example, by weakening, zeroing, or replacing the optical flow vector at that pixel location with a nominal optical flow vector, to reduce the impact of abnormal optical flow on subsequent processing.

[0052] Please see Figure 6 In some embodiments, step S104 includes, but is not limited to, steps S601 to S602: Step S601: Perform superpixel segmentation on the edge restoration image to obtain superpixel sub-images; Step S602: Perform temperature modeling on the superpixel sub-image to obtain a mixture of Gaussian thermal distribution models; Step S603: Dynamic segmentation threshold is generated based on the Gaussian heat distribution model and preset environmental thermal inertia parameters to obtain the confidence threshold. Step S604: Calculate the confidence level of the superpixel sub-image based on the confidence threshold to obtain the obstacle confidence map.

[0053] Steps S601 to S604 of this embodiment involve superpixel segmentation of the edge-restored image to obtain superpixel sub-images, thereby dividing the image into several local regions with similar features. This facilitates more refined image analysis within local areas. Based on this, temperature modeling is performed on the superpixel sub-images to obtain a Gaussian mixture thermal distribution model. This allows the temperature distribution characteristics of each superpixel region to be characterized through a statistical model, thereby improving the ability to distinguish between hot target regions and background regions. Furthermore, a dynamic segmentation threshold is generated based on the Gaussian mixture thermal distribution model and preset environmental thermal inertia parameters to obtain a confidence threshold. This allows the segmentation threshold to be adaptively adjusted in conjunction with environmental thermal change characteristics. Subsequently, confidence is calculated on the superpixel sub-images based on the confidence threshold to obtain an obstacle confidence map, thereby representing the probability that each region belongs to an obstacle in the form of probability or confidence. Thus, this embodiment improves the accuracy of determining potential obstacle regions and provides more reliable regional information for subsequent obstacle identification, thereby further enhancing the accuracy of obstacle identification.

[0054] In step S601 of some embodiments, a superpixel sub-image refers to a number of local region images formed by clustering pixels in the edge restoration image according to the principle of similarity. Each superpixel region consists of a group of adjacent pixels with similar characteristics, such as having similar gray values, temperature features, or spatial relationships.

[0055] In some implementations, clustering methods based on pixel similarity and spatial constraints can be used to perform superpixel segmentation on the edge restoration image. For example, adjacent pixels can be clustered according to their grayscale features, temperature features, and spatial relationships, so that pixels with similar features are grouped into the same superpixel region, thereby obtaining multiple superpixel regions, and each superpixel region is used as a superpixel sub-image.

[0056] In step S602 of some embodiments, the mixture Gaussian thermal distribution model refers to a statistical model that uses multiple Gaussian distribution functions to describe the temperature distribution characteristics within a superpixel region. Since the temperature distribution within the same superpixel region in an infrared image may be affected by various factors such as target thermal radiation, environmental background, and noise, a single statistical distribution is difficult to accurately describe its temperature characteristics. Therefore, a weighted combination of multiple Gaussian distributions can be used to model the temperature distribution of this region, thereby forming a mixture Gaussian thermal distribution model.

[0057] In some implementations, firstly, based on the mapping relationship between pixel grayscale values ​​and temperature in the infrared image, the pixel grayscale values ​​in the superpixel sub-image are converted into corresponding temperature values, thereby obtaining the temperature data of the superpixel region at the current moment; then, the temperature data is statistically analyzed, and multiple Gaussian distribution functions are used to fit the temperature data to obtain the mean, variance, and weight parameters of each Gaussian distribution, so as to form a mixed Gaussian thermal distribution model of the superpixel block at the current moment.

[0058] Please see Figure 7 In some embodiments, step S603 may include, but is not limited to, steps S701 to S703: Step S701: Perform mean statistics on the mixed Gaussian heat distribution model to obtain the mean value; Step S702: Perform variance statistics on the mixture Gaussian heat distribution model to obtain the variance value; Step S703: Calculate the confidence threshold based on the environmental thermal inertia parameters, mean value, and variance value.

[0059] Steps S701 to S703, as illustrated in this embodiment, involve performing mean statistics on the Gaussian mixture thermal distribution model to obtain a mean value, and performing variance statistics on the model to obtain a variance value. This allows for a quantitative description of the central trend and fluctuation degree of temperature distribution within the superpixel region, thereby reflecting the overall thermal characteristics and stability of the region. Based on this, a threshold is calculated using environmental thermal inertia parameters, the mean value, and the variance value to obtain a confidence threshold. This ensures that the confidence threshold not only reflects the temperature distribution characteristics of the current superpixel region but also dynamically adjusts to account for the influence of environmental thermal inertia on temperature changes. Through this method, the threshold calculation process can better conform to the actual changes in the infrared thermal field, thereby improving the accuracy of obstacle region determination, reducing the interference of complex thermal backgrounds on the recognition results, and ultimately enhancing the reliability and stability of obstacle recognition.

[0060] In some embodiments, steps S701 to S703 are performed to calculate the confidence threshold in the manner shown in equation (3): (3), among which, and For the first Each superpixel block at time Temperature statistical mean and variance , The adjustment coefficient is used; the integral term is used to characterize the accumulation and decay effect of heat in the background region over time. The time variable represents the integral process, and its range of values ​​is... to Used to indicate the current time Historical moments within a previous time window; The integral term represents the environmental thermal inertia time constant; it is used to characterize the accumulation of heat in the background area within the time window and the thermal inertia effect that decays over time; n represents the time window length parameter, used to determine the historical time range for participating in the thermal inertia accumulation calculation. This is the confidence threshold.

[0061] In step S604 of some embodiments, the obstacle confidence map refers to a probability distribution map used to represent the likelihood that each region in the image belongs to an obstacle target. Specifically, each pixel or superpixel region in the obstacle confidence map corresponds to a confidence value, which is used to represent the degree of probability that the region belongs to an obstacle target. The higher the confidence value, the greater the probability that the region belongs to an obstacle target; the lower the confidence value, the more likely the region belongs to the background region.

[0062] In some implementations, confidence can be calculated based on the relationship between the temperature characteristics of superpixel regions and a confidence threshold. For example, the average temperature value of each superpixel sub-image can be obtained first, and the average temperature value can be compared with the corresponding confidence threshold. When the difference between the average temperature value and the confidence threshold is large, the superpixel region is determined to be more likely to belong to a hot target region and is assigned a higher confidence value; when the difference between the average temperature value and the confidence threshold is small, the superpixel region is determined to be more likely to belong to a background region and is assigned a lower confidence value. Subsequently, the confidence values ​​of each superpixel region are mapped according to their spatial location in the image to generate a corresponding obstacle confidence map, which is used to characterize the distribution of potential obstacle regions in the entire image.

[0063] In step S105 of some embodiments, obstacle recognition refers to automatically detecting and identifying target areas in infrared images, and determining the category of target objects and their spatial location in the image.

[0064] In one embodiment, obstacle recognition can be achieved using a pre-trained neural network model. Specifically, the edge reconstruction image, the temperature-conserving optical flow field, and the obstacle confidence map can be input as multi-channel input features into the pre-trained neural network model, enabling the neural network model to output the obstacle category and obstacle location.

[0065] In one embodiment, during the training of the neural network, the loss function includes, but is not limited to, a physical consistency regularization term. Specifically, the process involves: first, obtaining the obstacle segmentation results output by the neural network, and extracting the corresponding segmentation boundaries based on the obstacle segmentation results; simultaneously, obtaining the physical edge information restored in step S102. Subsequently, spatial location matching is performed between the segmentation boundaries and the physical edges to obtain the geometric correspondence between the segmentation boundaries and the physical edges; finally, the model is updated and optimized based on the differences between these geometric correspondences.

[0066] Please see Figure 8 In one embodiment, the obstacle recognition device may further include a data acquisition module, a physical correction module, a mechanism-driven module, a statistical inference module, and an inference output module.

[0067] The data acquisition module is used to acquire infrared video streams and corresponding motion parameters, and to synchronously acquire and process the infrared video streams and motion parameters. The data acquisition module outputs the infrared image data and motion parameters to the physical correction module, and simultaneously outputs the motion parameters to the mechanism-driven module.

[0068] The physical correction module is used to restore the physical edges in the infrared image based on the infrared image and motion parameters. It corrects and enhances the boundary information in the infrared image using a physical imaging model, thereby obtaining an edge-restored image. This edge-restored image can more accurately reflect the contour information of real objects in the scene and serves as important input information for subsequent statistical inference processes.

[0069] The mechanism-driven module is used to establish a physical mechanism model related to scene changes based on motion parameters. This mechanism model is used to jointly model temperature changes and motion changes in the scene, thereby calculating a temperature-conserving optical flow field. This temperature-conserving optical flow field describes the dynamic relationship between temperature distribution and motion changes in an infrared image sequence.

[0070] The statistical inference module is used to jointly analyze the edge restoration image from the physical correction module and the temperature-conserving optical flow field from the mechanism-driven module. It uses statistical modeling methods to infer the probability of obstacle presence at different locations, thereby generating a confidence map. This confidence map characterizes the likelihood of obstacle presence at each spatial location.

[0071] The inference output module receives the confidence map output by the statistical inference module and combines it with the structural information in the edge restoration image to make a comprehensive judgment and inference analysis on the potential obstacle area. Finally, it outputs the obstacle category and obstacle location, realizing the identification and localization of obstacle targets in the scene.

[0072] Please see Figure 9 This application also provides an obstacle recognition device that can implement the above-described obstacle recognition method. The device includes: The data acquisition module 901 is used to acquire infrared images of the target object; The physical correction module 902 is used to perform edge repair on infrared images to obtain edge-restored images; The mechanism-driven module 903 is used to construct an optical flow field based on infrared images to obtain a temperature-conserving optical flow field. The statistical inference module 904 is used to extract confidence based on the edge restoration image to obtain the obstacle confidence map. The inference output module 905 is used to identify obstacles based on the obstacle confidence map, edge restoration image and temperature-conserving optical flow field, and obtain the obstacle category and obstacle location.

[0073] The specific implementation of this obstacle recognition device is basically the same as the specific implementation of the obstacle recognition method described above, and will not be repeated here.

[0074] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the obstacle recognition method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0075] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the obstacle recognition method of the embodiments of this application. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0076] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the obstacle recognition method described above.

[0077] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0078] The obstacle recognition method, device, electronic equipment, and storage medium provided in this application acquire infrared images of target objects and perform edge restoration processing on these images to obtain edge-restored images. This reduces the blurring or diffusion of target edges caused by thermal halo effects and complex thermal backgrounds in the infrared images, making the target outline clearer and facilitating accurate extraction of subsequent obstacle regions. Simultaneously, an optical flow field is constructed based on the infrared image to obtain a temperature-conserving optical flow field. This utilizes the stable characteristics of temperature attributes in the infrared image to constrain pixel motion, improving the reliability of motion information. Furthermore, confidence is extracted from the edge-restored image to obtain an obstacle confidence map, thereby improving the accuracy of potential obstacle region determination. Based on this, the obstacle confidence map, edge-restored image, and temperature-conserving optical flow field are fused for obstacle recognition, comprehensively utilizing image structure information, motion information, and region confidence information to improve the accuracy of obstacle recognition.

[0079] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0080] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0083] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0084] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0086] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An obstacle recognition method, characterized in that, The method includes: Acquire infrared images of the target object; The infrared image is subjected to edge restoration to obtain an edge-restored image; An optical flow field is constructed based on the infrared image to obtain a temperature-conserving optical flow field. Confidence is extracted from the edge reconstruction image to obtain an obstacle confidence map; Obstacle identification is performed based on the obstacle confidence map, the edge restoration image, and the temperature-conserving optical flow field.

2. The method according to claim 1, characterized in that, The process of edge restoration of the infrared image to obtain an edge-restored image includes: The infrared image is subjected to thermal target region detection to obtain a thermal target image; Based on the thermal target image, an adaptive point diffusion model is constructed to obtain the model. The infrared image is edge-restored by constructing the adaptive point diffusion model to obtain the edge-restored image.

3. The method according to claim 1, characterized in that, The step of extracting confidence scores from the edge reconstruction image to obtain an obstacle confidence map includes: The edge-restored image is subjected to superpixel segmentation to obtain superpixel sub-images; Temperature modeling is performed on the superpixel sub-image to obtain a mixture of Gaussian thermal distribution models; Based on the hybrid Gaussian heat distribution model and preset environmental thermal inertia parameters, a dynamic segmentation threshold is generated to obtain the confidence threshold. The confidence level of the superpixel sub-image is calculated based on the confidence threshold to obtain the obstacle confidence map.

4. The method according to claim 1, characterized in that, The step of constructing an optical flow field based on the infrared image to obtain a temperature-conserving optical flow field includes: The infrared image is subjected to vector calculation to obtain the optical flow vector; Temperature data is obtained by performing temperature calculations on the infrared image. The infrared image is subjected to grayscale calculation to obtain the image grayscale. The temperature-conserving optical flow field is obtained by calculating based on the optical flow vector, the temperature data, and the image grayscale.

5. The method according to claim 4, characterized in that, The process of constructing an optical flow field based on the infrared image to obtain a temperature-conserving optical flow field further includes: Obtain motion parameters; The nominal optical flow field is obtained by calculating based on the aforementioned motion parameters; The optical flow difference is obtained by performing pixel-by-pixel residual calculation based on the nominal optical flow field and the temperature-conserved optical flow field. If the optical flow difference is greater than a preset threshold, the temperature-conserving optical flow field is suppressed.

6. The method according to claim 2, characterized in that, The step of constructing an adaptive point diffusion model based on the thermal target image includes: Temperature is extracted from the thermal target image to obtain the target temperature difference; The distance to the target is obtained by performing distance calculation on the thermal target image; Based on the target temperature difference and the target distance, a model is constructed to obtain the adaptive point diffusion model, which is shown in the following equation: , The adaptive construction point diffusion model is as follows: , The target temperature difference, The target distance is... Atmospheric permeability, For the system parameters of the infrared camera, The coefficient of thermal stability. It is a basic diffusion term.

7. The method according to claim 3, characterized in that, The step of generating a confidence threshold based on the Gaussian heat distribution model and preset environmental thermal inertia parameters includes: The mean value was obtained by performing mean statistics on the mixed Gaussian heat distribution model. Variance statistics were performed on the mixture Gaussian heat distribution model to obtain the variance values; The confidence threshold is calculated based on the environmental thermal inertia parameters, the mean value, and the variance value, as shown in the following formula: , in, The confidence threshold is... For the first Each superpixel block in The mean value at time [time]. For the first The variance value of each superpixel block at time t. and For adjustment coefficients, To represent the time variable during the integration process, In order to be in Temperature difference over time This represents the time constant of environmental thermal inertia.

8. An obstacle recognition device, characterized in that, The device includes: The data acquisition module is used to acquire infrared images of the target object; The physical correction module is used to perform edge repair on the infrared image to obtain an edge-restored image; The mechanism-driven module is used to construct an optical flow field based on the infrared image to obtain a temperature-conserving optical flow field. The statistical inference module is used to extract confidence based on the edge restoration image to obtain an obstacle confidence map. The inference output module is used to identify obstacles based on the obstacle confidence map, the edge restoration image, and the temperature-conserving optical flow field, and to obtain the obstacle category and obstacle location.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the obstacle recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the obstacle recognition method according to any one of claims 1 to 7.

Images