Image processing method, apparatus and device based on semantic segmentation

Abstract

Embodiments of the present application provide a kind of based on semantic segmentation's image processing method, device and equipment. The method comprises: obtaining time sequence image sequence including target pipeline;According to time sequence image sequence, generate the feature image sequence of current period;Feature image sequence represents the temperature change rate information of each pixel point at different time in current period;According to feature image sequence, determine the moving direction of temperature anomaly region;According to time sequence image sequence, feature image sequence and the moving direction of temperature anomaly region, generate fusion feature map;Fusion feature map is used to represent the temperature value and temperature change rate information of each pixel point in current period by multiple channels, moving direction and channel have corresponding relationship;The semantic segmentation processing of fusion feature map is carried out, and the leakage area of target pipeline in current period is obtained. The method is used to reach the effect of improving the sensitivity of leakage detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image processing method and device based on semantic segmentation and an equipment. BACKGROUND

[0002] In the process of oil and gas production, transportation and storage, small leaks may occur in pipelines due to corrosion, fatigue or mechanical damage, resulting in energy loss, environmental pollution and even safety accidents. Therefore, it is necessary to detect the leakage of the pipeline.

[0003] In related technologies, feature extraction is usually performed based on single-frame static infrared images of the pipeline to identify whether leakage occurs. However, this method cannot capture dynamic features in the leakage process, resulting in a lag in early detection of small leaks, i.e., low sensitivity in leakage detection. SUMMARY

[0004] The embodiments of the present application provide an image processing method, device and equipment based on semantic segmentation to improve the sensitivity of leakage detection.

[0005] In a first aspect, the embodiments of the present application provide an image processing method based on semantic segmentation, comprising:

[0006] obtaining a time sequence image sequence including a target pipeline; wherein the time sequence image sequence represents temperature values of each pixel point at different times in a current period;

[0007] generating a feature image sequence of the current period according to the time sequence image sequence; wherein the feature image sequence represents temperature change rate information of each pixel point at different times in the current period;

[0008] determining a moving direction of a temperature abnormal area according to the feature image sequence;

[0009] generating a fusion feature map according to the time sequence image sequence, the feature image sequence and the moving direction of the temperature abnormal area; wherein the fusion feature map is used to represent temperature values and temperature change rate information of each pixel point in the current period through multiple channels, and the moving direction has a corresponding relationship with the channels;

[0010] performing semantic segmentation processing on the fusion feature map to obtain a leakage area of the target pipeline in the current period.

[0011] In a possible implementation, the generating of the fusion feature map according to the time sequence image sequence, the feature image sequence and the moving direction of the temperature abnormal area comprises:

[0012] determining a target direction category corresponding to the moving direction of the temperature abnormal area;

[0013] generate a heat map sequence of the current time period according to the target direction category and the feature image sequence; wherein each frame of the heat map sequence is used to represent temperature change rate information of each pixel point through a plurality of direction feature channels; the direction feature channel has a corresponding relationship with the target direction category;

[0014] generate a fusion feature map according to the heat map sequence and the time sequence image sequence.

[0015] In a possible implementation, the generating the fusion feature map according to the heat map sequence and the time sequence image sequence comprises:

[0016] determine a target direction feature channel from the plurality of direction feature channels of the heat map sequence;

[0017] generate a fusion feature map according to the target direction feature channel, the heat map sequence and the time sequence image sequence.

[0018] In a possible implementation, the determining the target direction feature channel from the plurality of direction feature channels of the heat map sequence comprises:

[0019] for each frame of the heat map sequence, determine an intensity value of each direction feature channel in the frame of the heat map;

[0020] determine the target direction feature channel according to the intensity value of each direction feature channel.

[0021] In a possible implementation, the generating the fusion feature map according to the target direction feature channel, the heat map sequence and the time sequence image sequence comprises:

[0022] for each pixel point in the time sequence image sequence, average temperature values of the pixel point at different time instants to obtain a pixel value of the pixel point in a first channel;

[0023] for each pixel point in the target direction feature channel, average temperature change rate information of the pixel point in the target direction feature channel in each frame of the heat map to obtain a pixel value of the pixel point in other channels; wherein the target direction feature channel corresponds to other channels one by one;

[0024] perform splicing processing on the first channel and the other channels to obtain the fusion feature map.

[0025] In a possible implementation, the generating the heat map sequence of the current time period according to the target direction category and the feature image sequence comprises:

[0026] For each pixel point of each frame of feature image in the sequence of feature images, if it is determined that the temperature change rate information of the pixel point in the sequence of N consecutive frames of feature images is greater than or equal to a first preset threshold, the temperature change rate information of the pixel point is amplified by a preset multiple to obtain a processed sequence of feature images; N is an integer greater than 1.

[0027] According to the processed sequence of feature images and the target direction category, a sequence of heat maps of the current period is generated.

[0028] In a possible implementation, the sequence of feature images also represents temperature gradient directions of each pixel point at different time instants of the current period; and the if it is determined that the temperature change rate information of the pixel point in the sequence of N consecutive frames of feature images is greater than or equal to the first preset threshold, the temperature change rate information of the pixel point is amplified by the preset multiple to obtain the processed sequence of feature images, includes:

[0029] If it is determined that the temperature change rate information of the pixel point in the sequence of N consecutive frames of feature images is greater than or equal to the first preset threshold, and an included angle between the temperature gradient direction of the pixel point and the preset target pipeline direction is less than or equal to a second preset threshold, the temperature change rate information of the pixel point is amplified by the preset multiple to obtain the processed sequence of feature images.

[0030] In a possible implementation, the determining a moving direction of the temperature abnormal region according to the sequence of feature images includes:

[0031] According to each frame of feature image in the sequence of feature images, a temperature abnormal region in each frame of feature image is determined.

[0032] First and second center of mass position information of the temperature abnormal region in adjacent two frames of feature images is determined.

[0033] According to the first and second center of mass position information, a moving direction of the temperature abnormal region is determined.

[0034] In a possible implementation, the determining a temperature abnormal region in each frame of feature image according to each frame of feature image in the sequence of feature images includes:

[0035] For each pixel point in the each frame of feature image, if it is determined that the temperature change rate information of the pixel point is greater than a seventh preset threshold, the pixel point is determined as an abnormal pixel point.

[0036] The abnormal pixel points in the each frame of feature image are subjected to connected region analysis to obtain the temperature abnormal region.

[0037] In a possible implementation, the semantic segmentation processing on the fusion feature map comprises:

[0038] The semantic segmentation processing on the fusion feature map comprises obtaining a segmentation image, wherein the segmentation image represents a probability value of each pixel belonging to a leakage region.

[0039] The semantic segmentation processing on the fusion feature map comprises obtaining a segmentation image, wherein the segmentation image represents a probability value of each pixel belonging to a leakage region.

[0040] In a possible implementation, the semantic segmentation processing on the fusion feature map comprises:

[0041] According to the segmentation image and a third preset threshold, a mask image is generated, wherein the third preset threshold is determined based on an actual pressure value of the target pipeline.

[0042] The semantic segmentation processing on the fusion feature map comprises obtaining a segmentation image, wherein the segmentation image represents a probability value of each pixel belonging to a leakage region.

[0043] If it is determined that the area of the initial leakage region is greater than a fourth preset threshold, the initial leakage region is taken as the leakage region of the target pipeline in the current period.

[0044] In a possible implementation, the fusion feature map further represents a temperature gradient direction of each pixel; and if it is determined that the area of the initial leakage region is greater than a fourth preset threshold, the initial leakage region is taken as the leakage region of the target pipeline in the current period, comprising:

[0045] If it is determined that the area of the initial leakage region is greater than a fourth preset threshold, a direction consistency index is determined according to the temperature gradient direction of the pixel included in the initial leakage region and a preset target pipeline direction.

[0046] If it is determined that the direction consistency index is greater than or equal to a fifth preset threshold, the initial leakage region is determined as the leakage region of the target pipeline in the current period.

[0047] In a possible implementation, the direction consistency index is determined according to the temperature gradient direction of the pixel included in the initial leakage region and a preset target pipeline direction, comprising:

[0048] For each pixel included in the initial leakage region, a cosine function value of a difference value between the temperature gradient direction of the pixel and the target pipeline direction is determined.

[0049] The cosine function values are averaged to obtain the direction consistency index.

[0050] In a possible implementation, the semantic segmentation processing on the fusion feature map comprises:

[0051] performing semantic segmentation processing on the fusion feature map to obtain an initial segmentation image; wherein the initial segmentation image represents an initial probability value of each pixel point belonging to a leakage region;

[0052] obtaining an ambient temperature of the target pipeline;

[0053] performing compensation processing on the initial probability value of each pixel point represented by the initial segmentation image according to the ambient temperature, to obtain the segmentation image.

[0054] In a possible implementation, the method further comprises:

[0055] obtaining a leakage region of the target pipeline in a historical period and a pressure gradient direction of the target pipeline;

[0056] determining a leakage area growth rate, third centroid position information of the leakage region in the historical period, and fourth centroid position information of the leakage region in the current period according to the leakage region in the historical period and the leakage region in the current period;

[0057] determining a moving direction of the leakage region according to the third centroid position information and the fourth centroid position information;

[0058] if it is determined that the leakage area growth rate is greater than a sixth preset threshold value, and an included angle between the moving direction of the leakage region and the pressure gradient direction is less than or equal to an eighth preset threshold value, outputting an alarm information.

[0059] In a second aspect, an image processing apparatus based on semantic segmentation is provided, comprising:

[0060] an obtaining module configured to obtain a time sequence image sequence comprising a target pipeline; wherein the time sequence image sequence represents temperature values of each pixel point at different time instants in a current period;

[0061] a generating module configured to generate a feature image sequence of the current period according to the time sequence image sequence; wherein the feature image sequence represents temperature change rate information of each pixel point at different time instants in the current period;

[0062] a determining module configured to determine a moving direction of a temperature abnormal region according to the feature image sequence;

[0063] a fusion module configured to generate a fusion feature map according to the time sequence images, the feature image sequence, and the moving direction of the temperature abnormal region, wherein the fusion feature map is configured to represent the temperature value and the temperature change rate information of each pixel point in a current time period through multiple channels, and the moving direction has a corresponding relationship with the channels;

[0064] a processing module configured to perform semantic segmentation processing on the fusion feature map to obtain a leakage region of the target pipeline in the current time period.

[0065] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor.

[0066] The memory stores computer execution instructions.

[0067] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0068] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0069] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0070] The image processing method, device and equipment based on semantic segmentation provided by the embodiments of the present application are as follows: an edge device acquires a time sequence image sequence containing a target pipeline, generates a feature image sequence bearing temperature change rate information based on the time sequence image sequence, determines the moving direction of a temperature abnormal region through the feature image sequence, then integrates the temperature value of the time sequence image, the temperature change rate of the feature image and the moving direction of the temperature abnormal region to generate a multi-channel fusion feature map, and finally performs semantic segmentation on the fusion feature map to obtain a leakage region. In this way, on the one hand, the limitations of traditional single-frame static infrared detection, which only relies on temperature static distribution and cannot capture dynamic characteristics of leakage, are broken through, and with the help of dynamic information of temperature change rate and abnormal region moving direction, subtle temperature fluctuations caused by small leakage can be identified earlier, the sensitivity of leakage detection is greatly improved, and early detection lag is avoided; on the other hand, through the fusion of multi-dimensional information such as temperature value, temperature change rate information and diffusion direction and the pixel-level positioning of semantic segmentation, environmental noise interference is effectively filtered, and the accuracy of leakage region positioning is improved. BRIEF DESCRIPTION OF DRAWINGS

[0071] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0072] Figure 1 A schematic diagram of an application scenario provided by the present application;

[0073] Figure 2 A schematic diagram of an image processing method based on semantic segmentation provided by the present application Figure 1 ;

[0074] Figure 3 A schematic diagram of an image processing method based on semantic segmentation provided by the present application Figure 2 ;

[0075] Figure 4 A schematic diagram of an image processing device based on semantic segmentation provided by the present application;

[0076] Figure 5 A schematic diagram of an electronic device provided by the present application.

[0077] The specific embodiments of the present application have been shown and described in the above-described drawings, and will be described in more detail hereinafter. These drawings and written description are not intended to restrict the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0078] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same elements throughout the several views. The embodiments described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0079] In the related art, features are mostly extracted based on single-frame static infrared images of the pipeline to identify whether leakage occurs. For example, patent CN119181097A proposes a near-infrared image semantic segmentation method based on BiSeNetV2, which synchronously extracts global context and local features through a bilateral network architecture, and adopts a mean precision evaluation model to evaluate the performance. However, this scheme focuses on static image processing and does not consider the time sequence evolution characteristics of the temperature in the pipeline leakage process, resulting in insufficient sensitivity to early-stage small leakage detection. Further, a multi-modal feature fusion method can be used to improve the sensitivity of leakage detection, for example, patent CN119380223A uses a ConvNeXt-T network to fuse RGB-D image features, and combines a pooling attention mechanism to enhance multi-modal data representation. However, this scheme introduces cross-modal attention to optimize feature fusion, but relies on visible light and depth information, which cannot directly adapt to the temperature field analysis requirements of single-channel infrared data, and does not analyze the spatio-temporal continuity of leakage diffusion.

[0080] Therefore, the inventors of the present application have conceived that the limitations of static single-frame data can be broken through by obtaining a time sequence image sequence, extracting the temperature dynamic change characteristics of each pixel point in the current period and the moving direction of the leakage diffusion from the time sequence temperature data, to perceive the slow temperature fluctuation caused by small leakage while further distinguishing between real leakage and environmental noise; and constructing a fusion feature map by multi-channel fusion of the static temperature value and the temperature dynamic change characteristics of the time sequence image, so that different channels carry different dimensional key information and the moving direction corresponds to the channel, ensuring the feature richness and relevance; finally, using semantic segmentation technology to process the fusion feature map to accurately distinguish the leakage area and the background, and ultimately achieving early-stage, high-sensitivity detection and accurate positioning of small leakage.

[0081] Figure 1 A schematic diagram of an application scenario provided by the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the leakage detection system includes a data acquisition device, an edge device, and a center device.

[0082] The data acquisition device is configured to acquire infrared images, pressure, environmental temperature, and other data of the target pipeline. The data acquisition device may, for example, be an infrared thermal imager array, an environmental temperature sensor, a pipeline pressure sensor, etc.

[0083] The edge device is configured to identify a leakage area based on the infrared images of the target pipeline. The edge device may, for example, be an electronic device with processing capability, such as a computer, a server, etc., and the present application embodiments do not limit the same. The execution subject of the present application embodiments may be the edge device.

[0084] The center device is configured to manage the leakage data reported by each edge device and determine whether to alarm. The center device may be a server or a server cluster.

[0085] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described again in some examples. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0086] Figure 2 Flowchart of the image processing method based on semantic segmentation provided by the present application Figure 1 As shown in the method comprises: Figure 2

[0087] S101, acquiring a time sequence image sequence comprising a target pipeline.

[0088] Among them, the time sequence image sequence represents the temperature value of each pixel point at different times in the current period.

[0089] Exemplarily, the target pipeline refers to a specific pipeline that needs to be detected for leakage, which can be a pipeline belonging to a key position prone to leakage in an oil and gas station, such as a pipeline segment of a node such as a bend, a valve, and a weld. The current period is a pre-set detection time interval, which can be 1 second, 5 seconds, etc., and can be set according to actual needs. The time sequence image sequence refers to a set of multiple frames of time sequence images continuously collected in time sequence within the current period, each frame of time sequence image can represent the temperature value of each pixel point on the surface of the pipeline at the corresponding time. In other words, the gray value of each pixel point in the time sequence image can represent the temperature value of the corresponding pipeline surface. The number of frames of the time sequence image sequence is not limited by the embodiments of the present application and can be set according to actual needs.

[0090] In one example, an infrared thermal imager array is arranged on the target pipeline, the thermal imager is triggered to work synchronously at a collection frame rate of 10 frames per second, the collection duration is set to 1 second (i.e. the current period is 1 second), the 10 frames of original infrared images collected are sorted by timestamp through the data transmission interface of the thermal imager, and the original infrared image sequence is generated and sent to the edge device. After the edge device receives the original infrared image sequence, it is calibrated to obtain the time sequence image sequence. Among them, the original infrared image represents the radiation intensity value of each pixel point.

[0091] Specifically, each frame of original infrared image in the original infrared image sequence can be calibrated by the following steps:

[0092] ​(1) Non-uniformity correction: in the initial calibration stage, the thermal imager collects a reference frame of T1=25℃ and a reference frame of T2=50℃ from the uniform blackbody radiation source; the gain coefficient and offset of each pixel are determined according to the following formulas 1 and 2 respectively; then, the original infrared image of each frame is corrected according to the following formula 3 to obtain the preliminary corrected image. Through this step, the radiation intensity value in the original infrared image can be converted into a temperature value, and the non-uniformity of the response of each pixel of the detector can be eliminated.

[0093] Formula 1

[0094] Formula 2

[0095] Formula 3

[0096] Wherein, G ij represents the gain coefficient of the pixel point (i, j); O ij represents the offset of the pixel point (i, j); V ij (T1) represents the pixel value of the pixel point (i, j) in the reference frame of T1=25℃; V ij (T2) represents the pixel value of the pixel point (i, j) in the reference frame of T2=50℃; V ij represents the pixel value of the pixel point (i, j) in the original infrared image; T ij represents the pixel value of the pixel point (i, j) in the preliminary corrected image. At this time, the pixel value of each pixel point in the preliminary corrected image is a temperature value.

[0097] (2) Bad pixel repair and data calibration: based on the preliminary corrected infrared image, the standard deviation and mean value of the temperature value of the same pixel between adjacent frames are determined, and the temperature abnormal bad pixel is determined based on the standard deviation and mean value, for example, the pixel point with a mean value greater than 3 times the standard deviation of the same pixel can be determined as a bad pixel, the temperature mean value of the 8-neighbor effective pixels around the bad pixel is replaced with the abnormal value by using the bilinear interpolation method, and the infrared image after bad pixel repair is obtained; after bad pixel repair, the environmental temperature value of the target pipeline is obtained, and the temperature value of each pixel point in the infrared image after bad pixel repair is corrected according to the Stefan-Boltzmann law (formula 4) to obtain the final corrected infrared image.

[0098] Formula 4

[0099] Wherein, represents the pixel value of each pixel point in the final corrected infrared image; represents the pixel value of each pixel point in the infrared image after bad pixel repair; represents the emissivity of the pipeline surface, which is a fixed material parameter of the pipeline; represents the ambient temperature value; represents the ambient emissivity (usually taken as 1).

[0100] (3) Temperature-gray scale mapping: A gray scale-temperature mapping lookup table is preset in the edge device, which is used to represent the corresponding relationship between the temperature value and the gray scale value, for example, gray scale value = k x temperature value + b; wherein, k and b are coefficients determined by blackbody furnace calibration experiment in advance. Further, the final corrected infrared image can be converted into a time sequence image through the gray scale-temperature mapping lookup table, at this time, the gray scale value of each pixel point of the time sequence image can represent the temperature value of the pipeline surface, which is convenient for subsequent image processing.

[0101] S102, generating a feature image sequence of the current period according to the time sequence image sequence.

[0102] Exemplarily, the feature image sequence refers to an image set focusing on the dynamic change characteristics of the pipeline temperature, which is generated by conversion based on the time sequence image sequence, and each frame of the time sequence image corresponds to a frame of the feature image. The feature image sequence is used to represent the temperature change rate information of each pixel point at different time in the current period, wherein the temperature change rate information is an index describing the change speed of temperature over time, which is used to represent the change amplitude of temperature per unit time. The temperature change rate information can intuitively reflect the dynamic fluctuation of the pixel point temperature, and provide a basis for subsequent identification of subtle temperature anomalies caused by leakage. The calculation accuracy of the temperature change rate is not limited in the embodiments of the present application, and can be adjusted according to the detection sensitivity requirement.

[0103] In one example, for the time sequence image sequence, a sliding window difference method is used to calculate the temperature change rate information, that is, taking two adjacent frames of time sequence images as a calculation window, for the same pixel point, the pixel value of the latter frame image is subtracted from the pixel value of the former frame image, and then divided by the time interval between the two frames to obtain the temperature change rate information of the pixel point; according to this method, all pixel points and adjacent frames are traversed to generate a feature image sequence, and each pixel value of each frame of the feature image is the temperature change rate information at the corresponding time.

[0104] In another example, t frames of time sequence images in the above time sequence image sequence are grouped into a three-dimensional matrix according to the time dimension, and a local space-time data block is constructed from the three-dimensional matrix using a sliding window wherein, x, y represent the coordinates of the pixel points in the time sequence image, and t represents the frame number; for example, a local space-time data block is constructed by 4 frames of time sequence images, and in the process of continuously sliding of the sliding window, convolution calculation processing can be performed on each local space-time data block based on a preset three-dimensional convolution kernel to obtain the temperature change rate information of each pixel point in each frame of time sequence image, that is, the feature image corresponding to each frame of time sequence image can be obtained, and then the feature image sequence can be obtained.

[0105] Specifically, the three-dimensional convolution kernel wherein, represents a first-order difference operator in the time dimension, which operates on data in the time dimension, can capture the trend of change in the time series by calculating the difference between the data at two time points and dividing by the time interval, and highlights the dynamic characteristics of the time dimension; represents a time offset, i.e. the moving step length of the first-order difference operator in the time dimension. For example, wherein, is the time interval between adjacent two time series images, taking the aforementioned current time period as 1 second and the time series image sequence including 10 time series images as an example, the first-order difference operator can be It can be understood that the first-order difference operator is equivalent to calculating the difference between a pixel point in the t+1 frame time series image and the pixel point in the t-1 frame time series image, and dividing by the time interval between the two time series images, i.e. , which conforms to the definition of the temperature change rate in the time dimension.

[0106] represents a convolution kernel in the spatial dimension, which can be, for example, The spatial dimension convolution kernel operates on pixel values in space, can capture the temperature gradient change of pixel points in the horizontal and vertical directions in the image, and highlights the edge or difference characteristics of the spatial dimension; , represents a spatial offset, i.e. the moving step length of the convolution kernel in the x, y direction in the spatial dimension.

[0107] If the pipeline as a whole slowly rises due to the ambient temperature (only the time dimension changes), at this time the time change rate of a single pixel will fluctuate, but the temperature change trend of the spatial neighborhood pixels is consistent (no local gradient), which belongs to normal environmental interference; when leakage occurs, the time dimension change rate of the leakage point pixel will quickly rise, and the surrounding pixels form a significant spatial temperature gradient due to heat diffusion (such as the leakage point temperature is high, and the adjacent pixel temperature is low). Therefore, if only the time dimension is considered, it will be mistaken that the environmental interference is leakage; and after combining the spatial dimension, through the weighted operation of the neighborhood pixels by the spatial dimension convolution kernel, the region with large time dimension change rate and significant spatial gradient can be identified, thereby accurately distinguishing the real leakage from the environmental interference.

[0108] Further, for each pixel point in each local space-time data block , the convolution calculation processing can be performed according to formula 5 to obtain the temperature change rate information of each pixel point in each time series image.

[0109] , formula 5

[0110] Referring to formula 5, for the local space-time data block Each position in The weights at the corresponding positions of the 3D convolution kernel Multiply by the pixel value at that location, then sum all the products to obtain the pixel value. Temperature change rate information in frame t This yields the feature image of frame t, which in turn allows us to obtain the feature image sequence. It should be noted that the temporal image of the boundary frames can be obtained by calculating the temperature change rate of each pixel using adjacent frames.

[0111] This method allows for the combination of spatial and temporal temperature changes based on three-dimensional convolution calculations, resulting in more accurate temperature change rate information for that pixel.

[0112] S103. Determine the direction of movement of the temperature anomaly region based on the feature image sequence.

[0113] For example, a temperature anomaly region refers to a connected region in a feature image sequence where the rate of temperature change exceeds a preset normal threshold. The direction of movement refers to the direction of positional shift of the temperature anomaly region in the feature image sequence over time, used to reflect the diffusion trend of the temperature anomaly region, and can help distinguish between real leakage (regular movement trend) and environmental interference (irregular movement).

[0114] In one example, as mentioned above, the pixel value of each pixel in the feature image represents the temperature change rate information. Therefore, the edge device can traverse each frame of the feature image sequence, mark pixels with temperature change rate information greater than a preset normal threshold as abnormal pixels, perform 8-neighborhood connected region analysis on the abnormal pixels in each frame of the feature image, and filter out continuous regions with a number of pixels greater than or equal to a preset number of pixels as the temperature abnormal region of that frame of the feature image; then, based on the feature images of two adjacent frames, calculate the centroid coordinates of the temperature abnormal region in the two frames respectively, subtract the centroid coordinates of the previous frame from the centroid coordinates of the later frame to obtain the displacement vector, and calculate the angle using the arctangent function according to the direction of the displacement vector to determine the movement direction of the temperature abnormal region.

[0115] S104. Generate a fused feature map based on the time-series image sequence, the feature image sequence, and the movement direction of the temperature anomaly region.

[0116] For example, a fused feature map refers to a multi-channel image that integrates the original temperature value of a time-series image sequence, the temperature change rate information of the feature image sequence, and the movement direction of the temperature anomaly region. In other words, the fused feature map is used to characterize the temperature value and temperature change rate information of each pixel in the current time period through multiple channels, and the movement direction has a corresponding relationship with the channel.

[0117] The multiple channels refer to independent information layers contained in the image, and different channels carry different types of features, such as one channel carrying temperature values and other channels carrying temperature change rates, and the moving direction has a corresponding relationship with the channel, that is, different moving directions correspond to different channels in the fused feature map, and each channel only retains the temperature change rate information in the corresponding direction. For example, the preset direction interval has a corresponding relationship with the channel, such as 0°≤moving direction<45° corresponding to the second channel, 45°≤moving direction<90° corresponding to the third channel, and the like. The number of channels is not limited in the embodiments of the present application, and can be set according to actual needs.

[0118] In one example, first, all frames of the time sequence image sequence are subjected to average processing to obtain an average temperature image representing the overall temperature distribution of the pipeline, which is taken as the first channel of the fused feature map, that is, carrying temperature values of each pixel point; then for each pixel point, if the moving direction of the temperature abnormal area to which the pixel point belongs belongs to a certain direction interval, the temperature change rate information of the pixel point is retained on the corresponding direction channel, and the pixel point value of the remaining direction channel is set to 0; further, for each direction channel, the pixel points of the direction channel are extracted from each frame of the feature sequence image and subjected to average processing to serve as other channels corresponding to the direction channel in the fused feature map, and each other channel and the first channel are spliced to form a multi-channel fused feature map, the first channel of the fused feature map carrying temperature values, and the other channels carrying temperature change rate information of different moving directions.

[0119] Optionally, the other channels in the fused feature map can carry temperature change rate information of all direction channels, or can carry temperature change rate information of part of the direction channels, which is not limited in the embodiments of the present application.

[0120] S105, performing semantic segmentation processing on the fused feature map to obtain a leakage area of the target pipeline in the current period.

[0121] Exemplarily, the semantic segmentation processing refers to classifying the pixel points in the fused feature map by using an image segmentation algorithm, and the classification target is a leakage area and a background area (i.e., a normal pipeline area without leakage). The leakage area refers to a continuous area formed by the pixel points determined as leakage after semantic segmentation, which corresponds to the actual leakage position of the pipeline, and the type of the image segmentation algorithm is not limited in the embodiments of the present application, for example, UNet, SegNet, and the like.

[0122] Taking an improved UNet network as the image segmentation algorithm as an example, the principle of the image segmentation algorithm is described.

[0123] The encoder part extracts multi-scale features of the input image using depthwise separable convolution. The encoder can include multiple levels, each level including a 3x3 depthwise convolution module, a 1x1 pointwise convolution module, a batch normalization and ReLU activation module, and a pooling module, which converts the input image into a feature map of different scales through layer-by-layer down-sampling.

[0124] The decoder part restores the image resolution using transpose convolution. The decoder corresponds to include multiple levels, each level including a 2x2 transpose convolution up-sampling module and a channel attention module, which gradually restores the high-dimensional semantic features extracted by the encoder to the original image resolution through layer-by-layer up-sampling. The channel attention module can perform global average pooling on the input feature map to generate a channel description vector; and calculate the channel weight coefficients through a two-layer fully connected network (including Sigmoid activation); multiply the weight coefficients with the original feature map channel by channel to realize dynamic weighting of features.

[0125] The improved UNet described above realizes multi-scale feature extraction through depthwise separable convolution of the encoder under lightweight calculation; the decoder restores the spatial resolution through transpose convolution and skip connection, and dynamically enhances key features using channel attention to suppress interference information. It can improve the running efficiency of edge devices through depthwise separable convolution, and enhance the sensitivity to weak leakage features through the channel attention mechanism, finally realize high-precision pixel-level segmentation of the pipeline leakage area, and adapt to the small leakage detection needs in complex environments of oil and gas stations.

[0126] In one example, the fused feature map is input into the improved UNet network described above, and the classification probability vector of each pixel point is output , ], respectively representing the probability that the pixel point (x, y) is background (i.e. non-leakage area) and leakage area. When is greater than or equal to the set probability threshold, the pixel point is determined to be a leakage pixel; all leakage pixels are analyzed for connected regions, and discrete small areas including less than a predetermined number of pixels are removed (to avoid noise interference), and the remaining continuous leakage pixel area is the target pipeline leakage area in the current period.

[0127] The image processing method based on semantic segmentation provided in the embodiments of the present application comprises the following steps: an edge device acquires a time sequence image sequence comprising a target pipeline, generates a feature image sequence carrying temperature change rate information based on the time sequence image sequence, determines a moving direction of a temperature abnormal area through the feature image sequence, integrates temperature values of the time sequence images, temperature change rates of the feature images and the moving direction of the temperature abnormal area, generates a multi-channel fusion feature map, and finally performs semantic segmentation on the fusion feature map to obtain a leakage area. In this way, on the one hand, the limitations of traditional single-frame static infrared detection, which only relies on temperature static distribution and cannot capture dynamic characteristics of leakage, are broken through. With the aid of dynamic information such as temperature change rate and abnormal area moving direction, subtle temperature fluctuations caused by small leaks can be identified earlier, the sensitivity of leakage detection is greatly improved, and early detection lag is avoided. On the other hand, through the fusion of multi-dimensional information such as temperature values, temperature change rate information and diffusion directions and the pixel-level positioning of semantic segmentation, environmental noise interference is effectively filtered, and the accuracy of leakage area positioning is improved.

[0128] Figure 3 Flowchart of the image processing method based on semantic segmentation provided in the present application Figure 2 As shown in Figure 3 , the embodiments of the present application are based on Figure 2 , the image processing method based on semantic segmentation is described in detail, which comprises the following steps:

[0129] S201, acquiring a time sequence image sequence comprising a target pipeline.

[0130] It should be noted that this step is similar to the aforementioned step S101 and will not be described here again.

[0131] S202, generating a feature image sequence of a current period according to the time sequence image sequence.

[0132] It should be noted that this step is similar to the aforementioned step S102 and will not be described here again.

[0133] S203, determining a temperature abnormal area in each frame of feature image according to each frame of feature image in the feature image sequence.

[0134] For example, for each pixel point in each frame of feature image, if it is determined that the temperature change rate information of the pixel point is greater than a seventh preset threshold value, the pixel point is determined as an abnormal pixel point. The abnormal pixel points in each frame of feature image are analyzed to obtain a temperature abnormal area.

[0135] The seventh preset threshold value is a critical value for distinguishing normal temperature fluctuation from abnormal temperature change, which can be determined based on empirical data and is a fixed value.

[0136] For example, taking the 5th frame of feature images in the sequence of feature images as an example: traversing all pixel points of the frame of feature images, for each pixel point (x, y), if the temperature change rate information of the pixel point > the seventh preset threshold, it is marked as an abnormal pixel point, otherwise it is marked as a normal pixel point, and finally the abnormal pixel points in the frame of feature images are obtained; using 8-neighborhood connected region analysis algorithm, that is, if two abnormal pixel points are adjacent in horizontal, vertical or diagonal direction, they are considered as the same region, clustering all abnormal pixel points to obtain multiple connected regions; setting the minimum area threshold to 5 pixels (regions smaller than the area are determined as noise), and removing the regions with an area smaller than the minimum area threshold, to obtain the temperature abnormal region in the frame of feature images.

[0137] S204, determining first and second center of mass position information of the temperature abnormal region in adjacent two frames of feature images.

[0138] Exemplarily, the adjacent two frames of feature images refer to two frames of feature images that are continuous in time, such as the t-1th frame and the tth frame. The first center of mass position information refers to the center of mass coordinates of the temperature abnormal region in the later frame (such as the tth frame) of feature images. The second center of mass position information refers to the center of mass coordinates of the temperature abnormal region in the former frame (such as the t-1th frame) of feature images. It can be understood that the center of mass coordinates can reflect the overall position of the temperature abnormal region, avoiding the interference of single pixel shift on direction judgment. The calculation method of the center of mass coordinates is not limited in the embodiments of the present application, and weighted average, geometric center method, etc. can be used. For example, for each pixel point included in the temperature abnormal region in the tth frame of feature images, the average value of the x-axis coordinates of the pixel points is taken as the x-axis coordinate value of the first center of mass position information, and the average value of the y-axis coordinates of the pixel points is taken as the y-axis coordinate value of the first center of mass position information, to obtain the first center of mass position information, and similarly obtain the second center of mass position information.

[0139] S205, determining the moving direction of the temperature abnormal region according to the first and second center of mass position information.

[0140] Exemplarily, the moving direction can be used to reflect the diffusion trend of the abnormal region, for example, the temperature abnormal region of real leakage usually has regular moving direction, such as along the pipeline direction, while the temperature abnormal region of environmental noise has irregular shift.

[0141] In one example, given the first center of mass position (x1, y1) and the second center of mass position information (x2, y2), the displacement vector dx = x1-x2, dy = y1-y2 is calculated; and the shift angle is calculated by the inverse tangent function to obtain the moving direction.

[0142] S206, determining the target direction category corresponding to the moving direction of the temperature abnormal region.

[0143] Exemplarily, the target direction category refers to a direction interval category to which the moving direction belongs. For example, a 360° circumference can be divided into several equal-angle intervals, such as 8 intervals, one interval every 45°, and each interval is a direction category. For example, the 8 direction categories can be represented by integers of 0 to 7, d=0 represents the moving direction , d=1 represents the moving direction , and so on.

[0144] Specifically, the target direction category d to which the moving direction belongs can be determined based on the moving direction by the following formula 6.

[0145] , formula 6

[0146] wherein d=0, 1, 2, …, 7.

[0147] It should be noted that the number of direction categories is not limited in the embodiments of the present application.

[0148] S207, generating a heat map sequence of the current time period according to the target direction category and the sequence of feature images.

[0149] Exemplarily, the heat map sequence is a multi-channel image set consistent with the number of frames of the sequence of feature images, and each frame of the sequence of feature images corresponds to a frame of the heat map. Each frame of the heat map sequence is used to represent the temperature change rate information of each pixel point through a plurality of direction feature channels; wherein the direction feature channel has a corresponding relationship with the target direction category.

[0150] The direction feature channel is an independent channel in the heat map that carries the temperature change rate information of a specific direction, and the number of channels is equal to the number of direction categories, such as 8 direction categories corresponding to 8 direction feature channels. The corresponding relationship between the direction feature channel and the target direction category means that each direction feature channel is bound to a target direction category, and the temperature change rate information belonging to the direction category is retained in the corresponding direction feature channel, and the values of other direction feature channels are set to 0.

[0151] In one example, as described above, the temperature abnormal area included in the feature image and the target direction category corresponding to the moving direction of the temperature abnormal area have been determined, and for each frame of the sequence of feature images, the corresponding 8-channel heat map is generated.

[0152] ​Specifically, the first channel (corresponding to the direction category 0) retains the temperature change rate information of the pixel points included in the temperature anomaly region with the target direction category d=0 of the moving direction, and the pixel points not belonging to the temperature anomaly region with the target direction category d=0 in the channel are set to 0; the second channel (corresponding to the direction category 1) retains the temperature change rate information of the pixel points included in the temperature anomaly region with the target direction category d=1 of the moving direction, and the pixel points not belonging to the temperature anomaly region with the target direction category d=1 in the channel are set to 0; and so on, to generate the 8-channel heat map corresponding to the frame feature image. In other words, the process of generating the heat map based on each frame feature image can be regarded as that the temperature change rate information of the pixel points in the temperature anomaly region included in the frame feature image is distributed to the direction feature channel corresponding to the moving direction. Through this step, the heat map sequence corresponding to the feature image sequence can be obtained.

[0153] In some possible implementations, for each pixel point of each frame feature image in the feature image sequence, if it is determined that the temperature change rate information of the pixel point in the consecutive N frames of feature images is greater than or equal to a first preset threshold, the temperature change rate information of the pixel point is amplified by a preset multiple to obtain a processed feature image sequence; and the heat map sequence of the current time period is generated according to the processed feature image sequence and the target direction category.

[0154] For example, N is an integer greater than 1; the first preset threshold is a critical value for determining whether the temperature change rate information needs to be amplified, and the first preset threshold is a fixed value set based on experience. The preset multiple is a ratio for amplifying the temperature change rate information, for example, can be 2 times, 3 times, etc., and can be set based on actual needs to enhance the saliency of abnormal features. The processed feature image sequence refers to a set of feature images processed by amplification, and the number of frames is consistent with the feature image sequence. It should be noted that the specific value of N, the size of the first preset threshold, and the value of the preset multiple are not limited in the embodiments of the present application.

[0155] For example, taking N=3 and the preset multiple being 2 times as an example, each pixel point of each frame feature image is traversed, the temperature change rate information of the pixel point in the previous 3 frames of feature images is determined, and if the temperature change rate information is greater than the first preset threshold, the temperature change rate information of the pixel point in the previous 3 frames of feature images is amplified by 2 times respectively; if not, the original value is kept unchanged to obtain the processed feature image sequence. Then, according to the foregoing manner, the corresponding 8-channel heat map is generated for each frame of the processed feature image in the processed feature image sequence to obtain the heat map sequence.

[0156] In this way, the saliency of the real leakage region (represented as a temperature anomaly in consecutive multiple frames) is enhanced, and the noise interference (abnormal pixels in a single frame) of isolated frames is suppressed.

[0157] In some possible implementation manners, the sequence of feature images can also represent temperature gradient directions of each pixel point at different time instants in the current time period. Specifically, the temperature gradient direction refers to a direction in which the temperature value changes most rapidly in space, and can reflect a local change trend of the pipeline surface temperature. A leakage area usually forms a radial temperature gradient with the leakage point as the center, and the temperature gradient of a normal area is relatively flat or irregular. Therefore, for each frame of time sequence image, a temperature gradient direction of each pixel point in the frame of time sequence image can be calculated through a preset mathematical operator. The mathematical operator can be, for example, a Sobel operator. For example, for each pixel point of each frame of time sequence image, a 3*3 neighborhood range of the pixel point is convolved based on the preset Sobel operator, to obtain a gradient component of the pixel point in an x direction and a gradient component of the pixel point in a y direction, and based on the gradient component in the x direction and the gradient component in the y direction, the temperature gradient direction of the pixel point can be obtained.

[0158] Further, if it is determined that the temperature change rate information of the pixel point in the sequence of feature images of the continuous N frames is greater than or equal to the first preset threshold, and an included angle between the temperature gradient direction of the pixel point and the preset target pipeline direction is less than or equal to a second preset threshold, the temperature change rate information of the pixel point is amplified by a preset multiple to obtain a processed sequence of feature images; and a sequence of heat maps of the current time period is generated according to the processed sequence of feature images and the target direction category.

[0159] Exemplarily, the preset target pipeline direction refers to a reference direction of the target pipeline, and can be set according to an actual laying direction of the pipeline. The included angle refers to an angle difference between the temperature gradient direction and the target pipeline direction. The second preset threshold is a critical angle for determining direction consistency, and can be set based on experience and is a fixed value.

[0160] For example, after determining that the temperature change rate information of the pixel point in the feature images of the continuous N frames is greater than or equal to the first preset threshold, it is simultaneously determined whether the included angle between the temperature gradient direction of the pixel point and the preset target pipeline direction is less than or equal to the second preset threshold. If both conditions are met, the temperature change rate information of the pixel point is magnified by a preset multiple. If there is a condition that is not met, the original value is kept unchanged. In this way, by using the time dimension condition that the temperature change rate of the continuous N frames meets the standard, the noise interference (such as abnormality caused by transient error of equipment) of a single frame is filtered. At the same time, by using the space dimension condition that the included angle between the temperature gradient direction and the pipeline direction meets the standard, the temperature anomaly (leakage usually spreads along the pipeline direction, and the gradient direction deviates from the pipeline direction) consistent with the pipeline direction is screened out. The feature magnification processing under the double conditions not only strengthens the dynamic features of the real leakage area, but also suppresses the random changes of environmental interference and the isolated anomalies irrelevant to the pipeline direction, thereby improving the accuracy of the thermograph sequence and the precision of the leakage detection.

[0161] S208, generating a fusion feature map according to the thermograph sequence and the time sequence image sequence.

[0162] Exemplarily, first, all frames of the time sequence image sequence are averaged to obtain an image capable of representing the average temperature. The image after the average processing is taken as the first channel of the fusion feature map. Then, from all 8-channel thermographs of the thermograph sequence, based on each direction feature channel, data of the direction feature channel in each frame of the thermograph is extracted. After the data of the direction feature channel is averaged, the average temperature change rate information of each pixel point of the direction feature channel is obtained as a channel of the fusion feature map. Therefore, a plurality of channels corresponding to each direction feature channel can be obtained. After the channels are spliced, the fusion feature map is obtained.

[0163] In some possible implementation manners, a target direction feature channel is determined from a plurality of direction feature channels of a thermograph sequence. A fusion feature map is generated according to the target direction feature channel, the thermograph sequence and the time sequence image sequence.

[0164] The target direction feature channel is a key channel highly related to the moving direction of the temperature anomaly area selected from the plurality of direction feature channels. The core role of the target direction feature channel is to focus on the direction feature most valuable for leakage detection and reduce redundant information interference. The number of the target direction feature channels selected and the selection basis are not limited in the embodiments of the present application. In other words, the target direction feature channel can be selected from the plurality of direction feature channels of the thermograph sequence. The data of the target direction feature channel and the time sequence image are spliced according to the channels to obtain the fusion feature map.

[0165] Specifically, for each frame of the heat map sequence, intensity values of each directional feature channel in the heat map are determined; target directional feature channels are determined according to the intensity values of each directional feature channel; for each pixel point in the time sequence image sequence, temperature values of the pixel point at different time instants are averaged to obtain a pixel value of the pixel point in the first channel; for each pixel point in the target directional feature channel, temperature change rate information of the pixel point in the target directional feature channel in each frame of the heat map sequence is averaged to obtain a pixel value of the pixel point in other channels; the first channel and the other channels are spliced to obtain a fused feature map. The target directional feature channel corresponds to the other channels one by one.

[0166] Exemplarily, the intensity value of the directional feature channel is an index for measuring the overall strength of the temperature change rate information in the single directional feature channel, is used to quantify the importance of the directional feature channel to the leakage detection, and can be obtained by statistical calculation of the temperature change rate information of all pixel points in the directional feature channel, for example, can be an average value, a sum, a maximum value, etc. The specific calculation method of the intensity value is not limited in the embodiment of the application. The first channel is a channel in the fused feature map that carries average temperature information of the pipeline surface, and is used to provide a spatial distribution benchmark of the temperature; the other channels are channels in the fused feature map corresponding to the target directional feature channel one by one, and are used to carry average temperature change rate information in different directions.

[0167] In one example, for each frame of the heat map sequence, an average value of the temperature change rate information of all pixel points in each directional feature channel in the frame of the heat map is taken as the intensity value of the directional feature channel; the intensity values of the directional feature channels in the frame of the heat map are sorted in descending order to obtain the first M candidate directional feature channels in the frame of the heat map, where M is an integer greater than 2; then a set of the first M candidate directional feature channels in each frame of the heat map can be obtained, the occurrence frequencies of the candidate directional feature channels in the set are determined, and the first M candidate directional feature channels with the highest occurrence frequencies are taken as the target directional feature channels. For example, the first directional feature channel, the second directional feature channel, the third directional feature channel, the fourth directional feature channel and the fifth directional feature channel are included in the first M candidate directional feature channels in the first, second, third, fourth and fifth frames of the heat map, and the occurrence frequencies are 5, 4 and 3 respectively. Therefore, the first directional feature channel, the second directional feature channel and the fourth directional feature channel are taken as the target directional feature channels.

[0168] Further, for each pixel point in the time sequence image sequence, the temperature values of the pixel point at different time instants are averaged to obtain a pixel value of the pixel point in the first channel, and all the pixel points are traversed to generate a complete first channel; for the target direction feature channel 1, the temperature change rate information of each pixel point in the target direction feature channel 1 in each frame of the heat map is averaged to obtain a pixel value of the pixel point in the second channel; similarly, pixel values of each pixel point in the third channel corresponding to the target direction feature channel 2 and pixel values of each pixel point in the fourth channel corresponding to the target direction feature channel 4 are obtained, that is, pixel values of each pixel point in other channels corresponding to the target direction feature channels are obtained, and the first channel and the other channels are spliced by an image channel splicing technology to obtain a fusion feature map. It should be noted that the above is an example of including three target direction feature channels, and a four-channel fusion feature map is obtained, and the number of channels of the fusion feature map is not limited in the embodiments of the present application.

[0169] Optionally, the temperature gradient direction of each pixel point at different time instants can also be included in the feature map sequence as described above. The temperature gradient direction of each pixel point can be averaged and added to the first channel of the fusion feature map or a separate channel to carry the temperature gradient direction of each pixel point. In other words, the fusion feature map can also represent the temperature gradient direction of each pixel point.

[0170] S209, performing semantic segmentation processing on the fusion feature map to obtain a segmentation image.

[0171] In some possible implementations, the fusion feature map is input into a preset image segmentation algorithm, and a segmentation image is output, which represents a probability value of each pixel point belonging to a leakage region.

[0172] In some possible implementations, the fusion feature map is subjected to semantic segmentation processing to obtain an initial segmentation image; the ambient temperature of the target pipeline is obtained; and according to the ambient temperature, an initial probability value of each pixel point represented by the initial segmentation image is compensated to obtain a segmentation image.

[0173] For example, the ambient temperature refers to the actual ambient air temperature (not the pipeline surface temperature) around the target pipeline, and the ambient temperature can affect the baseline value of the pipeline surface temperature and the temperature difference caused by the leakage. For example, in a high-temperature environment, the temperature difference between the pipeline and the leakage medium can be reduced, resulting in weakening of the leakage feature; and in a low-temperature environment, the temperature difference can be amplified. The initial segmentation image represents an initial probability value of each pixel point belonging to a leakage region.

[0174] Therefore, the fusion feature map can be input into a preset image segmentation algorithm to output an initial segmentation image; and based on the obtained ambient temperature of the target pipeline, the initial probability value of each pixel point represented by the initial segmentation image is compensated according to formula 7 below to obtain a segmentation image.

[0175] , formula 7

[0176] wherein, represents the probability value of a pixel point in the segmentation image; represents the initial probability value of a pixel point in the initial segmentation image; is a temperature attenuation coefficient, which is a fixed value; represents the ambient temperature; represents a theoretical temperature of a leakage point, which is a fixed value.

[0177] Through this step, the deviation of low probability in a low-temperature environment and high probability in a high-temperature environment is corrected based on the compensation processing, so that the probability value is more consistent with the actual leakage feature.

[0178] S210, performing connected region analysis on the segmentation image to obtain a leakage region of the target pipeline in the current period.

[0179] In some possible implementations, a mask image is generated according to the segmentation image and a third preset threshold; connected region analysis is performed on the mask image to obtain an initial leakage region; and if it is determined that the area of the initial leakage region is greater than a fourth preset threshold, the initial leakage region is taken as the leakage region of the target pipeline in the current period.

[0180] wherein, the mask image refers to an image obtained by performing binaryzation processing on the segmentation image, which only contains candidate leakage pixels (marked as 1) and background pixels (marked as 0), and the core function is to filter out the high-probability potential leakage region from the segmentation image to narrow the range for subsequent analysis. The third preset threshold is a probability threshold value for binaryzation, which can be a fixed value or determined based on the actual pressure value of the target pipeline.

[0181] It can be understood that the actual pressure value of the medium in the target pipeline (such as the operating pressure of the oil and gas pipeline) will affect the prominence of the leakage feature: under high pressure, the medium leakage speed is fast, the local temperature change is more intense, and the probability value of the leakage region is generally high, so the third preset threshold can be appropriately reduced (to avoid false negatives); under low pressure, the leakage feature is weak, the probability value is low, and the threshold needs to be appropriately increased (to avoid false positives). The actual pressure value of the target pipeline refers to the real-time pressure of the medium in the pipeline in the current period, which can be collected in real time by a pressure sensor.

[0182] The initial leakage region refers to a continuous pixel region after area screening, and the area of the initial leakage region refers to the number of pixels contained in the initial leakage region.

[0183] The fourth preset threshold is an area threshold for determining whether the initial leakage region is a real leakage region, and the value of the fourth preset threshold is based on the actual working condition of the target pipeline (such as pipeline diameter, medium type). It can be understood that the minimum leakage area of a large-diameter pipeline is usually larger than that of a small-diameter pipeline. Therefore, the fourth preset threshold can be set according to the pipeline size.

[0184] Exemplarily, the third preset threshold may be determined by , wherein, represents the reference threshold, which is a fixed value; represents the actual pressure value of the target pipeline; represents the maximum pressure bearing value of the target pipeline, which is a fixed value. Further, each pixel point of the segmented image is traversed, and if the leakage probability value of the pixel point is greater than or equal to the third preset threshold, the pixel point is marked as 1 (candidate leakage pixel); if the leakage probability value is less than the third preset threshold, the pixel point is marked as 0 (background pixel), and finally a binary mask image consistent with the size of the segmented image is obtained. Further, 8-neighborhood connected region analysis is performed on the mask image to obtain an initial leakage region; if it is determined that the area of the initial leakage region is greater than the fourth preset threshold, the initial leakage region is taken as the leakage region of the target pipeline in the current period.

[0185] In this way, the third preset threshold based on pressure correlation realizes dynamic adaptation of the threshold, avoids the problem of false negatives at high pressure and false positives at low pressure of the fixed threshold, and makes the selection of candidate leakage pixels more suitable for the leakage characteristics under the influence of medium pressure; and through the double screening of connected region analysis and area determination, discrete noise and small mislabeled regions are effectively eliminated, ensuring that the final leakage region has the continuity and area rationality of real leakage; at the same time, the fourth preset threshold adapts to the pipeline size, further improving the working condition adaptability of the result, and finally making the determination of the leakage region more accurate and reliable, and adapting to the leakage detection needs of target pipelines of different pressures and different diameters.

[0186] In some possible implementations, if it is determined that the area of the initial leakage region is greater than the fourth preset threshold, a direction consistency index is determined according to the temperature gradient direction of the pixel points included in the initial leakage region and the preset target pipeline direction; if it is determined that the direction consistency index is greater than or equal to a fifth preset threshold, the initial leakage region is determined as the leakage region of the target pipeline in the current period.

[0187] The direction consistency index is a parameter quantifying the degree of consistency of the temperature gradient direction of the pixel points in the initial leakage region with the target pipeline direction. The higher the proportion of pixels with consistent directions, the more likely it is a real leakage, because real leakage spreads along the pipeline, and the gradient direction is concentrated, while the gradient direction of the noise region is chaotic. The fifth preset threshold is a critical value for determining whether the direction consistency meets the characteristics of real leakage. The fifth preset threshold can be a fixed value set based on experience.

[0188] Optionally, for each pixel point included in the initial leakage region, a cosine function value of the difference between the temperature gradient direction of the pixel point and the target pipeline direction is determined. The cosine function values are averaged to obtain the direction consistency index. For example, the direction consistency index can be determined according to the following formula 8:

[0189] Formula 8

[0190] wherein, The direction consistency index is a parameter quantifying the degree of consistency of the temperature gradient direction of the pixel points in the initial leakage region with the target pipeline direction. The higher the proportion of pixels with consistent directions, the more likely it is a real leakage, because real leakage spreads along the pipeline, and the gradient direction is concentrated, while the gradient direction of the noise region is chaotic. The fifth preset threshold is a critical value for determining whether the direction consistency meets the characteristics of real leakage. The fifth preset threshold can be a fixed value set based on experience. The temperature gradient direction of the i-th pixel point is represented by The target pipeline direction is represented by

[0191] It can be understood that the cosine function value changes regularly when the angle difference between the gradient direction and the pipeline direction changes, which can exactly meet the requirement of direction consistency. For example, when the angle difference is 0°, it means that the gradient direction is consistent with the pipeline direction, and the cosine function value is 1, indicating that the directions are consistent. When the angle difference is 90°, it means that the gradient direction is perpendicular to the pipeline direction, and the cosine function value is 0, indicating that the directions are not consistent. When the angle difference is 180°, it means that the gradient direction is opposite to the pipeline direction, and the cosine function value is -1, indicating that the directions are completely opposite. Then, by averaging the cosine function values, the overall direction consistency of the initial leakage region can be reflected, avoiding abnormal interference of individual pixels and reducing the false detection rate of leakage detection.

[0192] Further, based on the foregoing formula 8, the direction consistency index can be determined according to the temperature gradient direction of the pixel points included in the initial leakage region and the preset target pipeline direction. If the direction consistency index is greater than or equal to the fifth preset threshold, the initial leakage region is determined as the leakage region of the target pipeline in the current period. In this way, the noise region with a qualified area but a chaotic direction can be filtered based on the direction consistency index, improving the accuracy of the leakage region.

[0193] In some embodiments, after determining the leakage area of the target pipeline in the current period, the edge device can generate a mask image of the leakage area in the current period, i.e., the pixel value of the leakage area is 1, and the pixel value of the non-leakage area is 0. The mask image of the leakage area is sent to the center device, and the center device can map each pixel point in the mask image of the leakage area to a three-dimensional position in the three-dimensional model of the pipeline based on the preset homography matrix H, and determine the color information corresponding to each pixel point based on the temperature change rate information of each pixel point, and display the leakage area in the current period in the three-dimensional model of the pipeline based on the color information of each pixel point. For example, a preset temperature change rate range and color correspondence relationship is determined, and then the color information of each pixel point can be determined based on the range to which the temperature change rate information of each pixel point belongs.

[0194] In some embodiments, the leakage area of the target pipeline in the historical period and the pressure gradient direction of the target pipeline are obtained; the leakage area growth rate, the third centroid position information of the leakage area in the historical period, and the fourth centroid position information of the leakage area in the current period are determined according to the leakage area in the historical period and the leakage area in the current period; the moving direction of the leakage area is determined according to the third centroid position information and the fourth centroid position information; if it is determined that the leakage area growth rate is greater than a sixth preset threshold, and the included angle between the moving direction of the leakage area and the pressure gradient direction is less than or equal to an eighth preset threshold, an alarm information is output.

[0195] For example, the above steps can be performed by the edge device or the center device, and the embodiments of the present application are not limited herein. The historical period refers to a preset detection period before the current detection period, for example, it can be 1 second. The leakage area in the historical period refers to the leakage area determined by the foregoing steps in the historical period. The pressure gradient direction of the target pipeline refers to the direction in which the medium in the pipeline flows from the high-pressure end to the low-pressure end (reflecting the natural tendency of medium diffusion, and the leakage usually expands along this direction), which can be calculated by collecting the pressure values at different positions of the pipeline, for example, by selecting two pressure monitoring points on the pipeline that are spaced apart by a preset distance, comparing the pressure values at the two points, and determining the direction from the high-pressure end to the low-pressure end as the pressure gradient direction.

[0196] The leakage area growth rate is an index for measuring the expansion speed of the leakage area, reflecting the severity of the leakage (the faster the rate, the higher the risk), which can be determined by the difference between the leakage area in the current period and the leakage area in the historical period, divided by the time interval between the two periods. The third centroid position information is the centroid coordinate of the historical leakage area, and the fourth centroid position information is the centroid coordinate of the current leakage area.

[0197] ​​​The moving direction of the leakage area refers to the overall offset direction of the leakage area from the historical period to the current period. The moving direction of the leakage area can be obtained by the coordinate difference between the fourth centroid position information and the third centroid position information, the displacement vector, and the arctangent function converting the displacement vector into an angle. The moving direction of the leakage area reflects the diffusion trend of the leakage with the medium flow or time elapse.

[0198] The sixth preset threshold is a critical value for determining whether the leakage area growth is too fast. The value can be set based on the type of pipeline medium and is a fixed value. The eighth preset threshold is an angle critical value for determining whether the moving direction of the leakage is consistent with the direction of the pressure gradient, and is a fixed value. The alarm information is a warning signal containing key information of the leakage (current area, growth rate, moving direction, and position coordinates), and is used to inform the operation and maintenance personnel to take measures. The specific values of the sixth preset threshold and the eighth preset threshold and the presentation form of the alarm information are not limited in the embodiments of the present application.

[0199] In one example, after the central device obtains the leakage area of the target pipeline in the historical period and the pressure gradient direction of the target pipeline, the central device determines the leakage area growth rate, the third centroid position information of the leakage area in the historical period, and the fourth centroid position information of the leakage area in the current period according to the leakage area in the historical period and the leakage area in the current period. Then, the central device determines the moving direction of the leakage area according to the third centroid position information and the fourth centroid position information, with reference to the calculation method of the moving direction of the temperature abnormal area. If it is determined that the leakage area growth rate is greater than the sixth preset threshold, and the included angle between the moving direction of the leakage area and the pressure gradient direction is less than or equal to the eighth preset threshold, the central device outputs the alarm information. For example, the alarm information can be a flashing prompt through the display interface of the central device, a sound and light alarm on site, automatic closing of the upstream valve and starting of emergency sprinkling, etc. The specific settings can be made according to actual needs.

[0200] Optionally, the alarm information can also be output when the leakage area growth rate is greater than the sixth preset threshold in a plurality of continuous periods, and the included angle between the moving direction of the leakage area and the pressure gradient direction in the plurality of periods is less than or equal to the eighth preset threshold.

[0201] In this way, on the one hand, the leakage area growth rate can reflect the expansion trend of the leakage, avoiding the limitation of only focusing on the current area and ignoring the dynamic risk (a small area but rapidly expanding leakage may be more dangerous than a large area slowly leaking); on the other hand, the leakage movement trend is verified in combination with the pressure gradient direction, ensuring that the alarm focuses on the high-risk leakage spreading along the medium flow direction (such leakage is easy to cause the range to rapidly expand due to the medium pushing, causing more serious leakage accidents). This alarm mechanism based on historical data and physical laws not only improves the accuracy of early warning, but also provides the dynamic characteristics of the leakage development for the operation and maintenance personnel, helps to develop targeted disposal schemes, effectively reduces the safety accidents and economic losses caused by pipeline leakage, and adapts to the operation and maintenance needs of key industrial facilities such as long oil and gas pipelines.

[0202] The image processing method based on semantic segmentation provided by the embodiments of the present application is that the edge device acquires a time sequence image sequence of a target pipeline, generates a feature image sequence containing temperature change rates; then determines a temperature abnormal area of each frame of the feature image, calculates a moving direction through the centroid positions of adjacent frame temperature abnormal areas, and classifies the moving direction into a target direction category; then generates a heat map sequence in combination with the target direction category and the feature image sequence, integrates the heat map sequence and the time sequence image sequence to obtain a fusion feature map; and finally performs semantic segmentation and connected region analysis on the fusion feature map to obtain a final leakage area. In this way, on the one hand, the multi-dimensional features of temperature static distribution, dynamic change rate and abnormal moving direction are integrated, which can accurately capture the spatio-temporal dynamic characteristics of the leakage, overcome the defect that the traditional single-frame detection is difficult to identify small leakage, and improve the early leakage detection sensitivity; on the other hand, through the direction category division and the connected region analysis, environmental noise interference is effectively filtered, false positives and false negatives are reduced, meanwhile, the multi-channel fusion provides sufficient information for semantic segmentation, realizes pixel-level accurate positioning of the leakage area, and meets the high-precision and high-reliability needs of pipeline safety monitoring in industrial scenes such as oil and gas stations.

[0203] Figure 4 The structure schematic diagram of the image processing device based on semantic segmentation provided by the present application is shown in Figure 4 The image processing device based on semantic segmentation provided by the embodiments of the present application 300 includes:

[0204] The acquisition module 301 is configured to acquire a time sequence image sequence including a target pipeline; wherein the time sequence image sequence represents the temperature value of each pixel point at different times in a current time period;

[0205] The generation module 302 is configured to generate a feature image sequence of the current time period according to the time sequence image sequence; wherein the feature image sequence represents the temperature change rate information of each pixel point at different times in the current time period;

[0206] The determining module 303 is configured to determine a moving direction of the temperature anomaly region according to the feature image sequence.

[0207] The fusion module 304 is configured to generate a fusion feature map according to the time sequence image sequence, the feature image sequence, and the moving direction of the temperature anomaly region, wherein the fusion feature map is used to represent the temperature value and the temperature change rate information of each pixel point in a current time period through multiple channels, and the moving direction has a corresponding relationship with the channels.

[0208] The processing module 305 is configured to perform semantic segmentation processing on the fusion feature map to obtain a leakage region of the target pipeline in the current time period.

[0209] In a possible implementation, the fusion module 304 is configured to:

[0210] determine a target direction category corresponding to the moving direction of the temperature anomaly region;

[0211] generate a heat map sequence of the current time period according to the target direction category and the feature image sequence, wherein each frame of the heat map sequence is used to represent the temperature change rate information of each pixel point through multiple direction feature channels, and the direction feature channels have a corresponding relationship with the target direction category;

[0212] generate the fusion feature map according to the heat map sequence and the time sequence image sequence.

[0213] In a possible implementation, the fusion module 304 is configured to:

[0214] determine a target direction feature channel from the multiple direction feature channels of the heat map sequence;

[0215] generate the fusion feature map according to the target direction feature channel, the heat map sequence, and the time sequence image sequence.

[0216] In a possible implementation, the fusion module 304 is configured to:

[0217] determine an intensity value of each direction feature channel in each frame of the heat map sequence;

[0218] determine the target direction feature channel according to the intensity value of each direction feature channel.

[0219] In a possible implementation, the fusion module 304 is configured to:

[0220] for each pixel point in the time sequence image sequence, perform average processing on temperature values of the pixel point at different time points to obtain a pixel value of the pixel point in the first channel;

[0221] For each pixel point in the target direction feature channel, the temperature change rate information of the pixel point in the target direction feature channel in each frame of the heat map is averaged to obtain the pixel value of the pixel point in other channels; the target direction feature channel corresponds to other channels one by one;

[0222] The first channel and the other channels are spliced to obtain a fused feature map.

[0223] In a possible implementation, the fusion module 304 is configured to:

[0224] For each pixel point in each frame of the feature image sequence, if it is determined that the temperature change rate information of the pixel point in the consecutive N frames of feature images is greater than or equal to a first preset threshold, the temperature change rate information of the pixel point is amplified by a preset multiple to obtain a processed feature image sequence; N is an integer greater than 1.

[0225] According to the processed feature image sequence and the target direction category, a heat map sequence of the current time period is generated.

[0226] In a possible implementation, the feature image sequence also represents the temperature gradient direction of each pixel point at different time instants in the current time period; the fusion module 304 is configured to:

[0227] If it is determined that the temperature change rate information of the pixel point in the consecutive N frames of feature images is greater than or equal to the first preset threshold, and the angle between the temperature gradient direction of the pixel point and the preset target pipeline direction is less than or equal to a second preset threshold, the temperature change rate information of the pixel point is amplified by a preset multiple to obtain a processed feature image sequence.

[0228] In a possible implementation, the determination module 303 is configured to:

[0229] According to each frame of the feature image sequence, a temperature abnormal area in each frame of the feature image is determined.

[0230] The first and second centroid position information of the temperature abnormal area in adjacent two frames of feature images is determined.

[0231] According to the first and second centroid position information, a moving direction of the temperature abnormal area is determined.

[0232] In a possible implementation, the determination module 303 is configured to:

[0233] For each pixel point in each frame of the feature image, if it is determined that the temperature change rate information of the pixel point is greater than a seventh preset threshold, the pixel point is determined as an abnormal pixel point.

[0234] The abnormal pixel points in each frame of feature image are subjected to connected region analysis to obtain a temperature abnormal region.

[0235] In a possible implementation, the processing module 305 is configured to:

[0236] The fusion feature map is subjected to semantic segmentation processing to obtain a segmentation image; the segmentation image represents a probability value of each pixel point belonging to a leakage region.

[0237] The segmentation image is subjected to connected region analysis to obtain a leakage region of the target pipeline in the current time period.

[0238] In a possible implementation, the processing module 305 is configured to:

[0239] According to the segmentation image and a third preset threshold, a mask image is generated; the third preset threshold is determined based on an actual pressure value of the target pipeline.

[0240] The mask image is subjected to connected region analysis to obtain an initial leakage region.

[0241] If it is determined that an area of the initial leakage region is greater than a fourth preset threshold, the initial leakage region is taken as the leakage region of the target pipeline in the current time period.

[0242] In a possible implementation, the fusion feature map further represents a temperature gradient direction of each pixel point; the processing module 305 is configured to:

[0243] If it is determined that an area of the initial leakage region is greater than a fourth preset threshold, a direction consistency index is determined according to a temperature gradient direction of a pixel point included in the initial leakage region and a preset direction of the target pipeline.

[0244] If it is determined that the direction consistency index is greater than or equal to a fifth preset threshold, the initial leakage region is determined as the leakage region of the target pipeline in the current time period.

[0245] In a possible implementation, the processing module 305 is configured to:

[0246] For each pixel point included in the initial leakage region, a cosine function value of a difference value between the temperature gradient direction of the pixel point and the direction of the target pipeline is determined.

[0247] The cosine function values are subjected to average processing to obtain the direction consistency index.

[0248] In a possible implementation, the processing module 305 is configured to:

[0249] The fusion feature map is subjected to semantic segmentation processing to obtain an initial segmentation image; the initial segmentation image represents an initial probability value of each pixel point belonging to a leakage region.

[0250] obtain an ambient temperature of the target pipeline;

[0251] According to the ambient temperature, the initial probability value of each pixel point represented by the initial segmentation image is compensated to obtain a segmentation image.

[0252] In a possible implementation, the apparatus further includes an alarm module configured to:

[0253] obtain a leakage area of the target pipeline in a historical period and a pressure gradient direction of the target pipeline;

[0254] According to the leakage area in the historical period and the leakage area in the current period, determine a leakage area growth rate, third centroid position information of the leakage area in the historical period, and fourth centroid position information of the leakage area in the current period;

[0255] According to the third centroid position information and the fourth centroid position information, determine a moving direction of the leakage area;

[0256] If it is determined that the leakage area growth rate is greater than a sixth preset threshold value, and an included angle between the moving direction of the leakage area and the pressure gradient direction is less than or equal to an eighth preset threshold value, output an alarm information.

[0257] The image processing apparatus based on semantic segmentation provided in this embodiment can execute the method provided in the method embodiment, and has similar implementation principles and technical effects, which will not be described here again.

[0258] Figure 5 The structure of the electronic device provided in this application is shown in FIG. 1. Figure 5 As shown in FIG. 1, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the electronic device 400 further includes a communication component 403. The processor 401, the memory 402, and the communication component 403 are connected through a bus. The electronic device can be the edge device or the center device.

[0259] In the specific implementation process, the at least one processor 401 executes the computer execution instructions stored in the memory 402, so that the at least one processor 401 executes the method described above.

[0260] The specific implementation process of the processor 401 can refer to the method embodiment, which has similar implementation principles and technical effects, and will not be described here again.

[0261] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0262] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0263] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0264] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above method.

[0265] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the above method is implemented.

[0266] The above readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0267] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0268] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0269] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0270] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0271] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0272] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various media capable of storing program codes, such as ROM, RAM, magnetic disk, or optical disk.

[0273] Finally, it should be noted that other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the present application disclosed herein. The present application is intended to include all such variations, uses, or adaptations of the application in which the general principles of the application are used to best advantage and encompassed within its scope. The present application is not limited to the precise structures described and shown in the accompanying drawings and figures, and can be practiced with variation of modifications and alterations without departing from the scope of the present application. The scope of the present application is limited only by the claims appended hereto.

Claims

1. An image processing method based on semantic segmentation, characterized in that, include: Acquire a time-series image sequence including the target pipeline; wherein the time-series image sequence represents the temperature value of each pixel at different times in the current time period; Based on the time-series image sequence, a feature image sequence for the current time period is generated; wherein, the feature image sequence represents the temperature change rate information of each pixel at different times in the current time period; Based on the feature image sequence, determine the direction of movement of the temperature anomaly region; A fused feature map is generated based on the time-series image sequence, the feature image sequence, and the movement direction of the temperature anomaly region; wherein, the fused feature map is used to characterize the temperature value and temperature change rate information of each pixel in the current time period through multiple channels, and the movement direction has a corresponding relationship with the channels; The fused feature map is semantically segmented to obtain the leakage area of ​​the target pipeline in the current time period; The step of generating a fused feature map based on the time-series image sequence, the feature image sequence, and the movement direction of the temperature anomaly region includes: Determine the target direction category corresponding to the movement direction of the temperature anomaly region; Based on the target direction category and the feature image sequence, a heatmap sequence for the current time period is generated; wherein, each frame of the heatmap sequence is used to characterize the temperature change rate information of each pixel through multiple directional feature channels; the directional feature channels have a corresponding relationship with the target direction category; A fused feature map is generated based on the heatmap sequence and the time-series image sequence.

2. The method according to claim 1, characterized in that, The step of generating a fused feature map based on the heatmap sequence and the time-series image sequence includes: From the multiple directional feature channels of the heatmap sequence, determine the target directional feature channel; A fused feature map is generated based on the target direction feature channel, the heatmap sequence, and the time-series image sequence.

3. The method according to claim 2, characterized in that, Determining the target directional feature channel from multiple directional feature channels of the heatmap sequence includes: For each frame of the heatmap sequence, determine the intensity value of each directional feature channel in that frame of the heatmap. The target direction feature channel is determined based on the intensity value of each directional feature channel.

4. The method according to claim 2, characterized in that, The step of generating a fused feature map based on the target direction feature channel, the heatmap sequence, and the time-series image sequence includes: For each pixel in the time-series image sequence, the average temperature value of the pixel at different times is used as the pixel value of the first channel; For each pixel in the target direction feature channel, the temperature change rate information of that pixel in the target direction feature channel in each frame of the heat map is averaged and used as the pixel value of that pixel in other channels; wherein, the target direction feature channel corresponds one-to-one with other channels; The first channel and the other channels are spliced ​​together to obtain the fused feature map.

5. The method according to claim 1, characterized in that, The step of generating the heatmap sequence for the current time period based on the target direction category and the feature image sequence includes: For each pixel in each frame of the feature image sequence, if it is determined that the temperature change rate information of the pixel is greater than or equal to a first preset threshold in the feature images of N consecutive frames, then the temperature change rate information of the pixel is amplified by a preset factor to obtain the processed feature image sequence; N is an integer greater than 1. Based on the processed feature image sequence and the target direction category, a heat map sequence for the current time period is generated.

6. The method according to claim 5, characterized in that, The feature image sequence also represents the temperature gradient direction of each pixel at different times in the current time period; if it is determined that the temperature change rate information of the pixel in the feature images of N consecutive frames is greater than or equal to a first preset threshold, then the temperature change rate information of the pixel is amplified by a preset factor to obtain the processed feature image sequence, including: If it is determined that the temperature change rate information of the pixel in the feature image of N consecutive frames is greater than or equal to the first preset threshold, and the angle between the temperature gradient direction of the pixel and the preset target pipe direction is less than or equal to the second preset threshold, then the temperature change rate information of the pixel is amplified by a preset factor to obtain the processed feature image sequence.

7. The method according to claim 1, characterized in that, Determining the direction of movement of the temperature anomaly region based on the feature image sequence includes: Based on each frame of the feature image sequence, determine the temperature anomaly region in each frame of the feature image; Determine the first centroid location information and the second centroid location information of the temperature anomaly region in two adjacent feature images; The direction of movement of the temperature anomaly region is determined based on the first centroid position information and the second centroid position information.

8. The method according to claim 7, characterized in that, The step of determining the temperature anomaly region in each frame of the feature image sequence based on each frame of the feature image sequence includes: For each pixel in each frame of the feature image, if the temperature change rate of the pixel is determined to be greater than the seventh preset threshold, then the pixel is determined to be an abnormal pixel. Connectivity analysis is performed on the abnormal pixels in each frame of the feature image to obtain the temperature anomaly region.

9. The method according to claim 1, characterized in that, The semantic segmentation process performed on the fused feature map to obtain the leakage area of ​​the target pipeline in the current time period includes: The fused feature map is subjected to semantic segmentation processing to obtain a segmented image; wherein, the segmented image represents the probability value of each pixel belonging to the leakage area; Connectivity analysis is performed on the segmented image to obtain the leakage area of ​​the target pipe in the current time period.

10. The method according to claim 9, characterized in that, The step of performing connected component analysis on the segmented image to obtain the leakage area of ​​the target pipe in the current time period includes: A mask image is generated based on the segmented image and a third preset threshold; wherein the third preset threshold is determined based on the actual pressure value of the target pipe. Connectivity analysis is performed on the mask image to obtain the initial leakage area; If the area of ​​the initial leakage region is determined to be greater than the fourth preset threshold, then the initial leakage region is taken as the leakage region of the target pipeline in the current time period.

11. The method according to claim 10, characterized in that, The fused feature map also characterizes the temperature gradient direction of each pixel; the step of determining that the area of ​​the initial leakage region is greater than a fourth preset threshold, and then using the initial leakage region as the leakage region of the target pipe in the current time period, includes: If it is determined that the area of ​​the initial leakage region is greater than the fourth preset threshold, then the directional consistency index is determined based on the temperature gradient direction of the pixels included in the initial leakage region and the preset target pipe direction. If the directional consistency index is determined to be greater than or equal to the fifth preset threshold, then the initial leakage area is determined as the leakage area of ​​the target pipeline in the current time period.

12. The method according to claim 11, characterized in that, The step of determining the directional consistency index based on the temperature gradient direction of the pixels included in the initial leakage area and the preset target pipe direction includes: For each pixel within the initial leakage area, determine the cosine function value of the difference between the temperature gradient direction of that pixel and the direction of the target pipe; The cosine function value is averaged to obtain the direction consistency index.

13. The method according to claim 9, characterized in that, The semantic segmentation process of the fused feature map to obtain a segmented image includes: The fused feature map is subjected to semantic segmentation processing to obtain an initial segmentation image; wherein, the initial segmentation image represents the initial probability value of each pixel belonging to the leakage area; Obtain the ambient temperature of the target pipeline; Based on the ambient temperature, the initial probability value of each pixel represented by the initial segmentation image is compensated to obtain the segmentation image.

14. The method according to any one of claims 1-13, characterized in that, The method further includes: Obtain the leakage area of ​​the target pipeline and the pressure gradient direction of the target pipeline during historical periods; Based on the leakage area in the historical period and the leakage area in the current period, determine the leakage area growth rate, the third centroid location information of the leakage area in the historical period, and the fourth centroid location information of the leakage area in the current period. The direction of movement of the leakage area is determined based on the third centroid position information and the fourth centroid position information; If it is determined that the rate of increase of the leakage area is greater than the sixth preset threshold, and the angle between the direction of movement of the leakage area and the direction of the pressure gradient is less than or equal to the eighth preset threshold, then an alarm message is output.

15. An image processing apparatus based on semantic segmentation, characterized in that, include: An acquisition module is used to acquire a time-series image sequence including a target pipeline; wherein the time-series image sequence represents the temperature value of each pixel at different times in the current time period; The generation module is used to generate a feature image sequence for the current time period based on the time-series image sequence; wherein the feature image sequence represents the temperature change rate information of each pixel at different times in the current time period; The determination module is used to determine the direction of movement of the temperature anomaly region based on the feature image sequence; The fusion module is used to generate a fused feature map based on the time-series image sequence, the feature image sequence, and the movement direction of the temperature anomaly region; wherein, the fused feature map is used to characterize the temperature value and temperature change rate information of each pixel in the current time period through multiple channels, and the movement direction has a corresponding relationship with the channels; The processing module is used to perform semantic segmentation processing on the fused feature map to obtain the leakage area of ​​the target pipeline in the current time period; The fusion module is specifically used to determine the target direction category corresponding to the movement direction of the temperature anomaly region; generate the heat map sequence for the current time period based on the target direction category and the feature image sequence; wherein, each frame of the heat map in the heat map sequence is used to characterize the temperature change rate information of each pixel through multiple directional feature channels; the directional feature channels have a corresponding relationship with the target direction category; and generate a fused feature map based on the heat map sequence and the time-series image sequence.

16. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-14.

Images