Image processing method and device, equipment, storage medium and computer program product

Abstract

The invention discloses an image processing method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: determining an overlapping region of a first three-dimensional image set and a second three-dimensional image set by using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, and deploying a first device at a cloud; using the overlapping region, the first feature set and the second feature set to obtain a third feature set, the third feature set including global features associated with the overlapping region; performing coding processing on the third feature set to obtain a pose matrix; the first three-dimensional image set and the second three-dimensional image set are registered through the pose matrix, a registered three-dimensional image is obtained, and the registered three-dimensional image is used for recognizing the abnormal condition of the first road. According to the technical scheme provided by the invention, prediction and feature refinement are carried out on the overlapping region, so that image registration is carried out based on more refined features, and the registration precision is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an image processing method, apparatus, device, storage medium and computer program product. Background Art

[0002] With the rapid development of intelligent transportation systems, three-dimensional images have become one of the irreplaceable effective information in the field of modern transportation. This is because: three-dimensional images can not only provide richer scene information, but also provide greater possibilities for traffic management and safety.

[0003] In order to realize the functions of intelligent transportation systems, three-dimensional image processing technology is particularly critical; among them, three-dimensional image registration technology can fuse three-dimensional image data generated at different time points, different angles or different sensors, so that the objects in each image accurately correspond to the same objects in other images; three-dimensional image recognition and detection technology can realize more comprehensive traffic scene analysis and intelligent decision-making.

[0004] However, related three-dimensional image registration solutions have the problem of strong dependence on the quality of three-dimensional image data, which makes it impossible to guarantee the accuracy of image registration. Summary of the invention

[0005] To solve the related technical problems, the embodiments of the present application provide an image processing method, apparatus, device, storage medium and computer program product.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The embodiment of the present application provides an image processing method, which is applied to a first device and includes:

[0008] Determine an overlapping area between the first three-dimensional image set and the second three-dimensional image set by using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, wherein the first device is deployed in the cloud;

[0009] Using the overlapping area, the first feature set and the second feature set, a third feature set is obtained, wherein the third feature set includes global features associated with the overlapping area;

[0010] Performing encoding processing on the third feature set to obtain a pose matrix;

[0011] The first three-dimensional image set and the second three-dimensional image set are registered using the pose matrix to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road.

[0012] In the above solution, the first three-dimensional image set includes M first image points, the second three-dimensional image set includes N second image points, and M and N are integers greater than or equal to 1; the method of obtaining the third feature set by using the overlapping area, the first feature set and the second feature set includes:

[0013] Determine first features for M first image points in the first feature set, and determine second features for N second image points in the second feature set;

[0014] Based on the self-attention mechanism, the first feature and the second feature are respectively updated using the overlapping area to obtain an updated first feature and an updated second feature;

[0015] The third feature set is obtained by using the updated first feature and the updated second feature.

[0016] In the above scheme, based on the self-attention mechanism, the first feature and the second feature are respectively updated by using the overlapping area to obtain the updated first feature and the updated second feature, including:

[0017] Based on the self-attention mechanism, determine a third feature and a fourth feature, wherein the third feature represents the degree of attention paid to a feature in the first feature associated with the overlapping area, and the fourth feature represents the degree of attention paid to a feature in the second feature associated with the overlapping area;

[0018] Using the third feature to update the first feature to obtain an updated first feature;

[0019] The second feature is updated using the fourth feature to obtain an updated second feature.

[0020] In the above scheme, the method further comprises:

[0021] determining one or more regions in the registered three-dimensional image, each region comprising one or more third image points;

[0022] Performing feature extraction on the third image points corresponding to each of the one or more regions to obtain one or more fifth features;

[0023] Performing fusion processing on the one or more fifth features to obtain a sixth feature;

[0024] Using the sixth feature, an abnormal situation on the first road is identified.

[0025] In the above scheme, the registered three-dimensional image includes L image layers, where L is an integer greater than or equal to 1; the feature extraction of the third image points corresponding to each of the one or more regions to obtain one or more fifth features includes:

[0026] At each image layer, feature extraction is performed on the third image points corresponding to each of the one or more regions to obtain one or more seventh features;

[0027] The one or more fifth features are obtained by using the one or more seventh features corresponding to the L image layers.

[0028] In the above scheme, the method further comprises:

[0029] Based on a dynamic graph convolutional neural network, extract features from the first three-dimensional image set to obtain the first feature set;

[0030] Based on a dynamic graph convolutional neural network, feature extraction is performed on the second three-dimensional image set to obtain the second feature set.

[0031] In the above solution, the first three-dimensional image set includes M first image points, where M is an integer greater than or equal to 1; the first three-dimensional image set is subjected to feature extraction based on the dynamic graph convolutional neural network to obtain the first feature set, including:

[0032] For each first image point, using the first image point and k first image points adjacent to the first image point, obtain a k nearest neighbor graph corresponding to the first image point, where k is an integer greater than or equal to 1;

[0033] Using k-nearest neighbor graphs corresponding to the M first image points, a first dynamic graph corresponding to the first three-dimensional image set is obtained, wherein edges in the first dynamic graph are determined based on relationships between different k-nearest neighbor graphs;

[0034] Based on the dynamic graph convolutional neural network, feature extraction is performed on the first dynamic graph to obtain the first feature set.

[0035] In the above scheme, the method further comprises:

[0036] A first task is received from a second device, wherein the first task is used to request registration of the first three-dimensional image set and the second three-dimensional image set, and the second device is a first device that determines to execute the first task based on a weight of the first road, wherein the weight of the first road is associated with a data size of the three-dimensional image and a complexity of the three-dimensional image.

[0037] The embodiment of the present application further provides an image processing apparatus, which is provided in a first device and includes:

[0038] A first determining unit is configured to determine an overlapping area between the first three-dimensional image set and the second three-dimensional image set by using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, wherein the first device is deployed in the cloud;

[0039] a second determining unit, configured to obtain a third feature set by using the overlapping area, the first feature set and the second feature set, wherein the third feature set includes global features associated with the overlapping area;

[0040] An encoding unit, used for encoding the third feature set to obtain a pose matrix;

[0041] A registration unit is used to register the first three-dimensional image set and the second three-dimensional image set using the posture matrix to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road.

[0042] The embodiment of the present application also provides a device, including: a processor and a communication interface; wherein,

[0043] The processor is used to determine the overlapping area of ​​the first three-dimensional image set and the second three-dimensional image set by using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, and the device is deployed in the cloud; using the overlapping area, the first feature set and the second feature set, a third feature set is obtained, and the third feature set includes global features associated with the overlapping area; encoding the third feature set to obtain a pose matrix; and using the pose matrix, registering the first three-dimensional image set and the second three-dimensional image set to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road.

[0044] The present application also provides a device, comprising: a processor and a memory for storing a computer program that can be run on the processor,

[0045] Wherein, the processor is used to execute the steps of any of the above-mentioned image processing methods when running the computer program.

[0046] An embodiment of the present application further provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned image processing methods.

[0047] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned image processing methods when executed by a processor.

[0048] The image processing method, apparatus, device, storage medium and computer program product provided by the embodiment of the present application, the first device uses the first feature set corresponding to the first three-dimensional image set and the second feature set corresponding to the second three-dimensional image set to determine the overlapping area of ​​the first three-dimensional image set and the second three-dimensional image set, and the first device is deployed in the cloud; the overlapping area, the first feature set and the second feature set are used to obtain a third feature set, and the third feature set contains global features associated with the overlapping area; the third feature set is encoded to obtain a pose matrix; the first three-dimensional image set and the second three-dimensional image set are registered using the pose matrix to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road. In the technical solution provided by the present application, the cloud server (i.e., the first device) generates global features with high correlation with the overlapping area by predicting and refining the features of the overlapping area between the two sets of three-dimensional image sets, so as to reduce the influence of non-overlapping areas on the global features. In this way, in the process of image registration based on global features, even if the image quality of the two sets of three-dimensional image collections is low, the same objects or contents in the overlapping areas can still be aligned based on more refined global features to achieve image registration. In this way, the accuracy of image registration can be guaranteed. In addition, on the basis of ensuring the accuracy of image registration, the reliability of subsequent anomaly detection and recognition results can also be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a flow chart of an image processing method according to an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a task allocation structure of an embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of the structure of a gesture encoder according to an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a foreign body identification method for the present application example;

[0053] Figure 5 A schematic diagram of the structure of a cloud server for three-dimensional image registration using an example of this application;

[0054] Figure 6 This is a schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0055] Figure 7 This is a schematic diagram of the device structure of an embodiment of the present application. DETAILED DESCRIPTION

[0056] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0057] In the related art, one 3D image registration scheme is to segment the point cloud data to correct the vertical and horizontal offsets respectively, and then use independent objects and road markings as registration primitives to achieve image registration. However, the recognition of road markings and independent objects is highly dependent on the quality and clarity of the 3D image data, which ignores the accuracy limitations of the equipment in the actual scene.

[0058] Based on this, in various embodiments of the present application, by predicting and refining the overlapping areas of the two sets of three-dimensional images, global features with high correlation with the overlapping areas in the three-dimensional images can be effectively extracted, thereby weakening the influence of non-overlapping areas on the global features. In this way, the accuracy of image registration based on global features can be guaranteed, and then the accuracy and reliability of anomaly detection based on the registered images can be guaranteed, so as to respond to foreign objects in complex road environments in a timely manner and ensure the safety of road traffic.

[0059] The present application embodiment provides an image processing method, such as Figure 1 As shown, applied to a first device, the method includes:

[0060] Step 101: using a first feature set corresponding to a first three-dimensional image set and a second feature set corresponding to a second three-dimensional image set, determining an overlapping area between the first three-dimensional image set and the second three-dimensional image set, wherein the first device is deployed in the cloud;

[0061] Step 102: using the overlapping area, the first feature set and the second feature set to obtain a third feature set, wherein the third feature set includes global features associated with the overlapping area;

[0062] Step 103: encoding the third feature set to obtain a pose matrix;

[0063] Step 104: using the pose matrix, registering the first three-dimensional image set and the second three-dimensional image set to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road.

[0064] In actual application, the first device can be understood as a server deployed in the cloud, that is, the first device can be called a cloud server; wherein, multiple first devices can be deployed in the cloud, for example, 5, and the embodiment of the present application does not limit the number of first devices.

[0065] At present, with the increasing complexity and diversity of urban road traffic, higher requirements are placed on the task allocation and scheduling capabilities of 3D image registration, and the task and resource allocation problem has gradually evolved from the original single-objective optimization to a multi-objective optimization problem. In the above case, the rationality of task allocation is related to the accuracy of 3D image registration and the accuracy of subsequent road anomaly detection. However, the related 3D image registration scheme has the problem of unreasonable task allocation, which leads to a high demand for resources (such as computing resources, memory resources) and computing costs when performing 3D image registration.

[0066] To address the above problem, before step 101, three-dimensional image registration tasks can be dynamically allocated to multiple first devices based on cloud service resource balancing technology to achieve reasonable task allocation, thereby maximizing resource utilization.

[0067] Based on this, in one embodiment, the method may further include:

[0068] A first task is received from a second device, wherein the first task is used to request registration of the first three-dimensional image set and the second three-dimensional image set, and the second device is a first device that determines to execute the first task based on a weight of the first road, wherein the weight of the first road is associated with a data size of the three-dimensional image and a complexity of the three-dimensional image.

[0069] In actual application, the second device can be understood as other servers other than the first device, and the second device can at least assign image registration tasks to multiple first devices based on a load balancing strategy. In addition, the first task can be understood as an image registration task under the first road scene.

[0070] In actual application, when there are p roads and q first devices (p and q are integers greater than or equal to 2), a set of three-dimensional image sets can be collected by the infrastructure (such as cameras) and vehicles on each road, and the second device determines the data size and complexity of the three-dimensional image corresponding to each road based on the collected three-dimensional image set, and then determines the weight of each road using the data size and complexity; wherein the weight of the road can be calculated using the following formula:

[0071]

[0072] Among them, W i It is expressed as the weight of the i-th road among p roads, It is represented as the 3D image collected from the infrastructure on the i-th road, It is represented as the set of 3D images collected by vehicles on the i-th road, and k is represented as the processing complexity of the 3D image set.

[0073] Next, based on the weighted round-robin algorithm (which can be expressed as Weight RoundRobin in English) in the load balancing strategy, the second device can use the weights of the p roads to determine the task ratio assigned to each first device for each road; wherein the task ratio can be determined by the following formula:

[0074]

[0075] Among them, δ i,j It is expressed as the task ratio of the i-th road assigned to the j-th first device among the q first devices, W total It is expressed as the sum of the weights of p roads, W total =∑W i , C j Denote the performance of the j-th first device.

[0076] In actual application, based on the determined task ratio, the second device can use the total number of tasks to determine the number of tasks assigned to each first device for each road (which can also be understood as the task size), and based on the determined number of tasks, send the first task to the first device, so that the first device can perform image registration corresponding to the first road; wherein the task size can be determined by the following formula:

[0077]

[0078] Here, Task(i,j) represents the number of tasks assigned to the jth first device for the i-th road.

[0079] For example, Figure 2 As shown, in the case where there are p roads and q first devices, the second device can, based on a load balancing strategy, sequentially assign tasks related to image registration to different first devices to ensure that the tasks are executed in a balanced manner according to road weights and device performance, thereby improving the efficiency of three-dimensional image registration.

[0080] In actual application, after receiving the first task, the first device can use a dynamic graph convolutional neural network (DGCNN) to extract local features of the first three-dimensional image set and the second three-dimensional image set respectively.

[0081] Based on this, in one embodiment, the method may further include:

[0082] Based on a dynamic graph convolutional neural network, extract features from the first three-dimensional image set to obtain the first feature set;

[0083] Based on a dynamic graph convolutional neural network, feature extraction is performed on the second three-dimensional image set to obtain the second feature set.

[0084] In actual application, the first three-dimensional image set may include one or more first three-dimensional images associated with the first road (which may be expressed as ), the first three-dimensional image set is collected by the infrastructure on the first road; the second three-dimensional image set may include one or more second three-dimensional images associated with the first road (which can be expressed as ), the second three-dimensional image set is collected by vehicles on the first road, and the number of the first three-dimensional images may be the same as the number of the second three-dimensional images.

[0085] In practical applications, the first three-dimensional image set and the second three-dimensional image set are three-dimensional data (also called three-dimensional point cloud data) composed of a large number of discrete image points (also referred to as points), which can describe the physical properties of the first road (such as geometric shape or surface information), but these properties are often irregular or unstructured, and there will be more noise interference or sparser data. In this case, through the dynamic graph convolutional neural network, features can be effectively extracted from the three-dimensional image data; wherein, the dynamic graph convolutional neural network can be understood as a neural network that can perform convolution on three-dimensional image data, and the basic idea is to realize the convolution of three-dimensional image data based on the k-nearest neighbor graph method.

[0086] Specifically, in one embodiment, the first three-dimensional image set includes M first image points, where M is an integer greater than or equal to 1; and the first three-dimensional image set is subjected to feature extraction based on the dynamic graph convolutional neural network to obtain the first feature set, including:

[0087] For each first image point, using the first image point and k first image points adjacent to the first image point, obtain a k nearest neighbor graph corresponding to the first image point, where k is an integer greater than or equal to 1;

[0088] Using k-nearest neighbor graphs corresponding to the M first image points, a first dynamic graph corresponding to the first three-dimensional image set is obtained, wherein edges in the first dynamic graph are determined based on relationships between different k-nearest neighbor graphs;

[0089] Based on the dynamic graph convolutional neural network, feature extraction is performed on the first dynamic graph to obtain the first feature set.

[0090] In practical applications, the k nearest neighbor graph corresponding to a first image point can be understood as a spherical structure formed by connecting the first image point with the adjacent (i.e., closest) k first image points. In addition, the k nearest neighbor graph can be called the neighborhood of the image point, which is not limited in the present embodiment.

[0091] In actual application, for the first three-dimensional image set, the first device can construct a k-nearest neighbor graph corresponding to M first image points in the Euclidean space through a k-nearest neighbor (KNN) algorithm; and construct edges in the first dynamic graph based on the interaction between the k-nearest neighbor graphs corresponding to the M first image points, thereby obtaining the first dynamic graph.

[0092] Here, after obtaining the first dynamic graph, the first device can determine the local spatial relationship between different first image points in the first dynamic graph based on the dynamic graph convolutional neural network, and then obtain the features corresponding to each first image point, that is, the first feature set can include the features of each first image point in the M first image points; wherein, edge convolution (which can be expressed as EdgeConv in English) can be used to determine the local spatial relationship between the first image points, so that the extracted features can contain more structural information. In this way, feature extraction of the first dynamic graph is achieved;

[0093] For example, in the process of edge convolution, it is assumed that the feature of the first image point a in the first layer is The first device may be configured to Calculate the features of the next layer Among them, the following formula can be used to calculate the features of the next layer:

[0094]

[0095] Among them, K a It is expressed as the sum of the k first image points, h θ () indicates input The difference between the features of each adjacent point b is used to capture the local spatial relationship between different first image points and output the feature vector corresponding to the a-th first image point. It can be implemented by a multilayer perceptron (MLP) with the parameter θ. max() means processing the feature vectors between different first image points through the maximum pooling operation and selecting the feature vector with the largest value for each feature channel. Since the maximum pooling operation can usually retain the most significant features of the image, it helps to effectively distinguish the structural information of the image through significant features such as the image contour, and increases the robustness to small noise.

[0096] Next, in order to transfer structural information between different layers, the first device may connect the feature vectors of each layer to obtain the final feature vector of the a-th first image point: Among them, the final feature vector can be obtained using the following formula:

[0097]

[0098] Among them, cat() means connecting the feature vectors. They are represented as the feature vectors of the 1st, 2nd, 3rd and 4th layers respectively.

[0099] Similarly, when the second three-dimensional image set contains N second image points (N is an integer greater than or equal to 1), for each second image point, the first device can use the second image point and k first image points adjacent to the second image point to obtain a k-nearest neighbor graph corresponding to the second image point; use the k-nearest neighbor graph corresponding to the N second image points to obtain a second dynamic graph corresponding to the second three-dimensional image set, and the edges in the second dynamic graph are determined based on the relationship between different k-nearest neighbor graphs; based on the dynamic graph convolutional neural network, determine the local spatial relationship between different second image points in the second dynamic graph, and then obtain the features corresponding to each second image point, that is, the second feature set can include features corresponding to each second image point among the N second image points.

[0100] In actual application, in step 101, the first device may respectively predict corresponding overlapping areas based on the interaction between the first feature set and the second feature set.

[0101] Specifically, for the first feature set, based on the attention mechanism, the first device may define a first query vector (which may be expressed as Among them, W Q is the learnable weight matrix, represented as matrix multiplication, denoted as the feature vector of the ath first image point in the first feature set), the first key vector (which can be represented as W K is a learnable weight matrix) and the first value vector (which can be expressed as W V is a learnable weight matrix), thereby calculating the corresponding attention score; wherein, the attention score can be calculated using the following formula:

[0102]

[0103] Among them, d k Represented as the dimension of the first key vector; in addition, the number of heads of multi-head attention can be 4.

[0104] Based on the result of formula (6), the first device can determine the feature vectors that need to be strengthened in the M first image points, and fuse the determined feature vectors with the feature vectors of the N second image points (which can also be understood as increasing the weight of the overlapping area and reducing the weight of the non-overlapping area) to update the first feature set. The updated first feature set can be expressed as:

[0105] φ X =F X +transformer(F X ,F Y ) (7)

[0106] Among them, F X It is represented as a set of feature vectors of the M first image points, F Y It is represented as a set of feature vectors of N second image points.

[0107] Next, using the updated first feature set, the first device may predict the probability of each first image point in the overlapping area; wherein the probability may be calculated using the following formula:

[0108] S = softmax(W P ·φ X ) (8)

[0109] Where S represents the probability of the first image point in the overlapping area; W P It is represented as a learnable weight matrix, which can be implemented through a one-dimensional convolutional layer, and the output dimension is 1.

[0110] Then, the first device may generate a first feature set capable of describing the structural distribution of the potential overlapping area based on formula (8); wherein the first feature set capable of describing the structural distribution of the potential overlapping area may be expressed as:

[0111] φ X * =softmax(W P ·φ X )·φ X =S·φ X (9)

[0112] Similarly, for the second feature set, the first device may respectively define a second query vector, a second key vector, and a second value vector, so as to calculate the corresponding attention score; using the obtained attention score, the first device may determine the feature vectors that need to be strengthened in the N second image points, and fuse the determined feature vectors with the feature vectors of the M first image points to update the second feature set. The updated second feature set may be expressed as:

[0113] φ Y =F Y +transformer(F Y ,F X ) (10)

[0114] Next, using the updated second feature set, the first device can predict the probability of each second image point in the overlapping area, and use the obtained probability to generate a second feature set that can describe the structural distribution of the potential overlapping area.

[0115] That is, in the process of determining the overlapping area, the first device may mix some features in the first feature set and the second feature set to adjust the first feature set and the second feature set respectively, so that the adjusted first feature set and the second feature set can describe the overlapping area.

[0116] In actual application, since the image points in the overlapping area may be unevenly distributed, the first device can use a self-attention mechanism to operate on the features of the image points, so that the network layer can selectively focus on the overlapping area based on the feature relationship between the image points. This helps to improve the performance of the network layer in tasks that require capturing long-range dependencies and global information, so as to obtain more accurate global features.

[0117] Specifically, in one embodiment, the obtaining a third feature set by using the overlapping area, the first feature set and the second feature set includes:

[0118] Determine first features for M first image points in the first feature set, and determine second features for N second image points in the second feature set;

[0119] Based on the self-attention mechanism, the first feature and the second feature are respectively updated using the overlapping area to obtain an updated first feature and an updated second feature;

[0120] The third feature set is obtained by using the updated first feature and the updated second feature.

[0121] The first feature may be understood as a feature set with the first image point as a dimension, and the second feature may be understood as a feature set with the second image point as a dimension.

[0122] In actual application, the first device may determine the first feature from the adjusted first feature set, and determine the second feature from the adjusted second feature set.

[0123] Here, in the process of updating the first feature and the second feature, the first device may update the first feature and the second feature through a self-attention feature.

[0124] Specifically, in one embodiment, based on the self-attention mechanism, the first feature and the second feature are respectively updated using the overlapping area to obtain the updated first feature and the updated second feature, including:

[0125] Based on the self-attention mechanism, determine a third feature and a fourth feature, wherein the third feature represents the degree of attention paid to a feature in the first feature associated with the overlapping area, and the fourth feature represents the degree of attention paid to a feature in the second feature associated with the overlapping area;

[0126] Using the third feature to update the first feature to obtain an updated first feature;

[0127] The second feature is updated using the fourth feature to obtain an updated second feature.

[0128] In actual application, for the first feature, by inputting the first feature into two convolutional layers, the first device can obtain a third value vector (which can be expressed as W c is a learnable weight matrix) and the third query vector (which can be expressed as W d is a learnable weight matrix); using the third value vector and the third query vector, the first device can calculate the third feature; by combining the third feature and the first feature, the first device can update the first feature; wherein the third feature can be calculated by the following formula:

[0129]

[0130] In addition, the first feature can be updated by the following formula:

[0131]

[0132] Among them, α is represented as a hyperparameter, which is used to control the contribution of the third feature to the first feature.

[0133] In actual application, for the second feature, by inputting the second feature into two convolutional layers, the first device can obtain a fourth value vector and a fourth query vector; using the fourth value vector and the fourth query vector, the first device can calculate the fourth feature; by combining the fourth feature and the second feature, the first device can update the second feature.

[0134] Here, after completing the update of the first feature and the second feature, the first device can obtain the global features associated with the overlapping area, that is, the third feature set; wherein the global features associated with the overlapping area have the characteristics of high feature similarity and small difference.

[0135] In actual application, in step 103, the first device can encode the global features associated with the overlapping area through a posture encoder to regress the pose matrix (also called transformation matrix, conversion matrix or pose parameter matrix, etc.) between the first three-dimensional image set and the second three-dimensional image set.

[0136] It should be noted that in the field of deep learning, Figure 3 As shown in the figure, a pose encoder usually refers to a neural network model that encodes input data (such as an image or a feature vector) into a latent space representation. In the latent space feature representation, the difference features between similar sample images will be eliminated as non-core information, and only the core features will be retained. Therefore, after the features are mapped to the latent space, the distance between similar features will be closer.

[0137] In actual application, the first device can encode the global features associated with the overlapping area through a posture encoder, so that the latent space can learn the posture information of the first three-dimensional image set and the second three-dimensional image set to the greatest extent, thereby accurately regressing the pose matrix (which can be expressed as (R, T)).

[0138] Next, using the posture matrix, the first device can transform the first three-dimensional image set so that the transformed first three-dimensional image set has the same spatial coordinates as the second three-dimensional image set; and splice the transformed first three-dimensional image set and the second three-dimensional image set to obtain a registered three-dimensional image.

[0139] In actual application, in order to detect anomalies on the first road, the first device can divide the registered three-dimensional image into different local areas, and extract features in each local area in turn. Through this step-by-step processing method, the perception range of the registered three-dimensional image can be gradually expanded, and more detailed and comprehensive features can be gradually extracted from the local to the global range.

[0140] Based on this, in one embodiment, the method may further include:

[0141] determining one or more regions in the registered three-dimensional image, each region comprising one or more third image points;

[0142] Performing feature extraction on the third image points corresponding to each of the one or more regions to obtain one or more fifth features;

[0143] Performing fusion processing on the one or more fifth features to obtain a sixth feature;

[0144] Using the sixth feature, an abnormal situation on the first road is identified.

[0145] Among them, the fifth feature can be understood as a local feature corresponding to the region, and the sixth feature can be understood as a global feature obtained by summarizing different local features.

[0146] In actual application, the first device can divide the one or more regions in the registered three-dimensional image through PointNet++; wherein PointNet++ is a neural network for data processing that performs feature learning on unstructured three-dimensional point cloud images and has the characteristics of a simple structure.

[0147] Here, in order to extract spatial information of different scales, the first device may use a layered sampling technique to extract features from each region.

[0148] Specifically, in one embodiment, the registered three-dimensional image includes L image layers, where L is an integer greater than or equal to 1; and extracting features from the third image points corresponding to each of the one or more regions to obtain one or more fifth features includes:

[0149] At each image layer, feature extraction is performed on the third image points corresponding to each of the one or more regions to obtain one or more seventh features;

[0150] The one or more fifth features are obtained by using the one or more seventh features corresponding to the L image layers.

[0151] The seventh feature may be understood as a local feature corresponding to a region in the image layer.

[0152] In actual application, for the first image layer among L image layers, the first device can use the seventh feature corresponding to the first image layer and the association between the third image point in the first image layer and the adjacent k third image points to obtain the seventh feature corresponding to the second image layer; that is, the first device can use the features of the current image layer to extract the features of the next image layer until the feature extraction of L image layers is completed, thereby obtaining the one or more fifth features.

[0153] Exemplarily, for the lth image layer among the L image layers, by querying the k adjacent third image points corresponding to each third image point, the first device can obtain an adjacency matrix (which can be expressed as A l =KNN(P l ,k),P l Represented as the three-dimensional image data of the l-th image layer); using the adjacency matrix and the seventh feature F of the l-th image layer l , extract features from each region in the l+1th image layer, and obtain the seventh feature of the l+1th image layer (which can be expressed as F l+1 =g(F l ,A l )).

[0154] Here, after completing the feature extraction of the Lth image layer, the first device may aggregate one or more fifth features corresponding to each region to obtain the sixth feature, so as to enhance the expression ability of the feature; wherein the sixth feature may be expressed as: global =max(F l+1 ), max() represents the maximum pooling operation on all features in the L-th image layer.

[0155] Then, through the fully connected network, the first device can classify the sixth feature and determine the abnormal situation on the first road according to the classification result; wherein the sixth feature can be classified by the following formula:

[0156] Y=h(F global ) (13)

[0157] Among them, h() represents a fully connected network, which can map the sixth feature to a classification label, so as to achieve the purpose of identifying abnormal situations on the first road.

[0158] In the image processing method provided by the embodiment of the present application, the first device determines the overlapping area of ​​the first three-dimensional image set and the second three-dimensional image set using the first feature set corresponding to the first three-dimensional image set and the second feature set corresponding to the second three-dimensional image set, and the first device is deployed in the cloud; the third feature set is obtained using the overlapping area, the first feature set and the second feature set, and the third feature set contains global features associated with the overlapping area; the third feature set is encoded to obtain a pose matrix; the first three-dimensional image set and the second three-dimensional image set are registered using the pose matrix to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road. In the technical solution provided by the present application, the cloud server (i.e., the first device) generates global features with high correlation with the overlapping area by predicting and refining the features of the overlapping area between the two sets of three-dimensional image sets, so as to reduce the influence of non-overlapping areas on the global features. In this way, in the process of image registration based on global features, even if the image quality of the two sets of three-dimensional image collections is low, the same objects or contents in the overlapping areas can still be aligned based on more refined global features to achieve image registration. In this way, the accuracy of image registration can be guaranteed. In addition, on the basis of ensuring the accuracy of image registration, the reliability of subsequent anomaly detection and recognition results can also be guaranteed.

[0159] The present application is described in further detail below in conjunction with application examples.

[0160] The application example of this application proposes a three-dimensional image recognition and detection solution for foreign objects on the road based on cloud computing big data; specifically, on the one hand, through cloud service resource balancing technology, tasks are dynamically allocated according to load conditions; on the other hand, a dynamic graph convolutional neural network is used to extract local features of the three-dimensional image data respectively, and the feature information of the two sets of three-dimensional images is mixed to predict possible overlapping areas. Then, the self-attention mechanism is used to effectively capture global information, and the posture encoder is used to regress the position transformation matrix of the two sets of three-dimensional images to achieve three-dimensional image registration. Finally, the registered three-dimensional image is used to identify foreign objects on the road.

[0161] Here, in the process of identifying foreign objects on the road, Figure 4 As shown, the following steps are included:

[0162] Step 401: Collecting three-dimensional images of road infrastructure and vehicles;

[0163] Specifically, for each road, the road infrastructure and the vehicle respectively collect three-dimensional images of the road to obtain two sets of three-dimensional images (ie, the first three-dimensional image set and the second three-dimensional image set mentioned above).

[0164] Step 402: The electronic device (i.e., the second device mentioned above) implements load balancing;

[0165] Specifically, based on the size and processing complexity of the three-dimensional image data, the electronic device calculates the performance weight of each section of road, and uses a weighted polling algorithm to calculate the task ratio of each section of road data assigned to each cloud server, thereby achieving the purpose of load balancing.

[0166] Step 403: the electronic device performs task allocation;

[0167] Here, the electronic device sends an image processing task (ie, the first task mentioned above) to the cloud server (ie, the second device mentioned above); after receiving the image processing task, the cloud server performs computing power configuration.

[0168] Step 404: The cloud server performs three-dimensional image registration;

[0169] Here, if Figure 5 As shown, the cloud server may include a feature extraction module, an overlapping area prediction module, a posture estimation module and a recognition detection module; wherein the three-dimensional image registration process may be implemented by the above four modules.

[0170] First, for each group of three-dimensional images, the feature extraction module uses a dynamic graph convolutional neural network to connect each point in the three-dimensional image with its k neighboring points to form a k-nearest neighbor graph; a dynamic graph is constructed using the k-nearest neighbor graphs corresponding to multiple points, and the edges in the dynamic graph are determined based on the interactions between different k-nearest neighbor graphs; based on the dynamic graph convolutional neural network, a convolution operation is performed on the dynamic graph to extract the features of the three-dimensional image (i.e., the first feature set and the second feature set mentioned above).

[0171] Secondly, the overlapping area prediction module uses the attention mechanism of Transformer to predict the overlapping area of ​​two sets of 3D images; the features of the 3D images can replace the position embedding information in the Transformer, and the updated features can be used to predict the probability of each point in the overlapping area.

[0172] Next, the posture estimation module uses the self-attention mechanism to operate on each point, that is, combining the self-attention features (i.e., the third and fourth features mentioned above) with other features to obtain the global features (i.e., the third feature set mentioned above); based on the global features, the posture encoder module is used to encode the global features of the areas with high overlap probability of the two sets of three-dimensional images, and the posture parameter matrix is ​​regressed. Thus, the posture parameter matrix is ​​used to perform precise geometric transformation to obtain a complete three-dimensional point cloud image after splicing and registration.

[0173] Step 405: three-dimensional image recognition detection;

[0174] Specifically, first, through PointNet++, the recognition and detection module groups the input registered complete 3D point cloud data into local areas, and then extracts local features in each local area. Through step-by-step processing, the recognition and detection module can gradually expand the perception range, gradually extract more detailed and comprehensive features from local to global, and input them into the fully connected network to output label probabilities through the softmax activation function classifier.

[0175] Step 406: the recognition detection module determines whether there is a foreign object based on the tag probability;

[0176] Wherein, when the output label probability is less than or equal to the set threshold (the value can be set as needed), it is determined that there is no foreign matter and step 407 is executed; when the output label probability is greater than the set threshold, it is determined that there is a foreign matter and step 408 is executed;

[0177] Step 407: No foreign object is detected on the road, and the vehicle is notified to pass normally;

[0178] Step 408: Issue an alarm to the vehicle-side big data and road big data.

[0179] In the application example of this application, firstly, through cloud service resource balancing technology, computing resources are dynamically allocated and released according to the load conditions to ensure that computing resources are maximized. In this way, efficient data processing load balancing is achieved, ensuring efficient processing in a big data environment.

[0180] Secondly, by applying DGCNN and Transformer structures, local and global features in 3D image data are effectively extracted, and the influence of non-overlapping areas on global features is weakened, thereby greatly improving the registration accuracy and ensuring the accuracy of subsequent foreign body detection. In addition, PointNet++ gradually downsamples 3D point cloud images and extracts features at each level, further enhancing the robustness of the model.

[0181] Third, the use of a posture encoder to generate some corresponding point pairs avoids the need to process a large number of corresponding point pairs and also saves computing resources.

[0182] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an image processing device, which is arranged on a first device, such as Figure 6 As shown, the device comprises:

[0183] A first determining unit 601 is configured to determine an overlapping area between the first three-dimensional image set and the second three-dimensional image set by using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, wherein the first device is deployed in the cloud;

[0184] A second determining unit 602 is configured to obtain a third feature set by using the overlapping area, the first feature set and the second feature set, wherein the third feature set includes global features associated with the overlapping area;

[0185] The encoding unit 603 is used to encode the third feature set to obtain a pose matrix;

[0186] The registration unit 604 is used to register the first three-dimensional image set and the second three-dimensional image set using the posture matrix to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road.

[0187] In one embodiment, the first three-dimensional image set includes M first image points, the second three-dimensional image set includes N second image points, and M and N are integers greater than or equal to 1; the second determining unit 602 is used to:

[0188] Determine first features for M first image points in the first feature set, and determine second features for N second image points in the second feature set;

[0189] Based on the self-attention mechanism, the first feature and the second feature are respectively updated using the overlapping area to obtain an updated first feature and an updated second feature;

[0190] The third feature set is obtained by using the updated first feature and the updated second feature.

[0191] In one embodiment, the second determining unit 602 is configured to:

[0192] Based on the self-attention mechanism, determine a third feature and a fourth feature, wherein the third feature represents the degree of attention paid to a feature in the first feature associated with the overlapping area, and the fourth feature represents the degree of attention paid to a feature in the second feature associated with the overlapping area;

[0193] Using the third feature to update the first feature to obtain an updated first feature;

[0194] The second feature is updated using the fourth feature to obtain an updated second feature.

[0195] In one embodiment, the device may further include: an identification unit; the identification unit is used to:

[0196] determining one or more regions in the registered three-dimensional image, each region comprising one or more third image points;

[0197] Performing feature extraction on the third image points corresponding to each of the one or more regions to obtain one or more fifth features;

[0198] Performing fusion processing on the one or more fifth features to obtain a sixth feature;

[0199] Using the sixth feature, an abnormal situation on the first road is identified.

[0200] In one embodiment, the registered three-dimensional image includes L image layers, where L is an integer greater than or equal to 1; and the recognition unit is used to:

[0201] At each image layer, feature extraction is performed on the third image points corresponding to each of the one or more regions to obtain one or more seventh features;

[0202] The one or more fifth features are obtained by using the one or more seventh features corresponding to the L image layers.

[0203] In one embodiment, the first determining unit 601 is further configured to:

[0204] Based on a dynamic graph convolutional neural network, extract features from the first three-dimensional image set to obtain the first feature set;

[0205] Based on a dynamic graph convolutional neural network, feature extraction is performed on the second three-dimensional image set to obtain the second feature set.

[0206] In one embodiment, the first three-dimensional image set includes M first image points, where M is an integer greater than or equal to 1; the first determining unit 601 is used to:

[0207] For each first image point, using the first image point and k first image points adjacent to the first image point, obtain a k nearest neighbor graph corresponding to the first image point, where k is an integer greater than or equal to 1;

[0208] Using k-nearest neighbor graphs corresponding to the M first image points, a first dynamic graph corresponding to the first three-dimensional image set is obtained, wherein edges in the first dynamic graph are determined based on relationships between different k-nearest neighbor graphs;

[0209] Based on the dynamic graph convolutional neural network, feature extraction is performed on the first dynamic graph to obtain the first feature set.

[0210] In one embodiment, the first determination unit 601 is further used to receive a first task sent by a second device, the first task is used to request alignment of the first three-dimensional image set and the second three-dimensional image set, and the second device is a first device that performs the first task based on the weight of the first road, and the weight of the first road is associated with the data size of the three-dimensional image and the complexity of the three-dimensional image.

[0211] In actual application, the first determination unit 601 can be implemented by a processor in the image processing device in combination with a communication interface; the second determination unit 602, encoding unit 603, registration unit 604 and recognition unit can be implemented by a processor in the image processing device.

[0212] It should be noted that: when the image processing device provided in the above embodiment performs image processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0213] Based on the hardware implementation of the above program module, and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a device (ie, the first device mentioned above), such as Figure 7 As shown, the device 700 includes:

[0214] Communication interface 701, capable of exchanging information with other devices;

[0215] A processor 702, connected to the communication interface 701 to implement information interaction with other devices, and used to execute the method provided by one or more of the above technical solutions when running a computer program;

[0216] A memory 703 , on which the computer program is stored.

[0217] Specifically, the processor 702 is used to determine the overlapping area of ​​the first three-dimensional image set and the second three-dimensional image set using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, and the device is deployed in the cloud; using the overlapping area, the first feature set and the second feature set, a third feature set is obtained, and the third feature set contains global features associated with the overlapping area; encoding the third feature set to obtain a pose matrix; and using the pose matrix, registering the first three-dimensional image set and the second three-dimensional image set to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road.

[0218] In one embodiment, the first three-dimensional image set includes M first image points, the second three-dimensional image set includes N second image points, and M and N are integers greater than or equal to 1; the processor 702 is configured to:

[0219] Determine first features for M first image points in the first feature set, and determine second features for N second image points in the second feature set;

[0220] Based on the self-attention mechanism, the first feature and the second feature are respectively updated using the overlapping area to obtain an updated first feature and an updated second feature;

[0221] The third feature set is obtained by using the updated first feature and the updated second feature.

[0222] In one embodiment, the processor 702 is configured to:

[0223] Based on the self-attention mechanism, determine a third feature and a fourth feature, wherein the third feature represents the degree of attention paid to a feature in the first feature associated with the overlapping area, and the fourth feature represents the degree of attention paid to a feature in the second feature associated with the overlapping area;

[0224] Using the third feature to update the first feature to obtain an updated first feature;

[0225] The second feature is updated using the fourth feature to obtain an updated second feature.

[0226] In one embodiment, the processor 702 is further configured to:

[0227] determining one or more regions in the registered three-dimensional image, each region comprising one or more third image points;

[0228] Performing feature extraction on the third image points corresponding to each of the one or more regions to obtain one or more fifth features;

[0229] Performing fusion processing on the one or more fifth features to obtain a sixth feature;

[0230] Using the sixth feature, an abnormal situation on the first road is identified.

[0231] In one embodiment, the registered three-dimensional image includes L image layers, where L is an integer greater than or equal to 1; the processor 702 is configured to:

[0232] At each image layer, feature extraction is performed on the third image points corresponding to each of the one or more regions to obtain one or more seventh features;

[0233] The one or more fifth features are obtained by using the one or more seventh features corresponding to the L image layers.

[0234] In one embodiment, the processor 702 is further configured to:

[0235] Based on a dynamic graph convolutional neural network, extract features from the first three-dimensional image set to obtain the first feature set;

[0236] Based on a dynamic graph convolutional neural network, feature extraction is performed on the second three-dimensional image set to obtain the second feature set.

[0237] In one embodiment, the first three-dimensional image set includes M first image points, where M is an integer greater than or equal to 1; the processor 702 is configured to:

[0238] For each first image point, using the first image point and k first image points adjacent to the first image point, obtain a k nearest neighbor graph corresponding to the first image point, where k is an integer greater than or equal to 1;

[0239] Using k-nearest neighbor graphs corresponding to the M first image points, a first dynamic graph corresponding to the first three-dimensional image set is obtained, wherein edges in the first dynamic graph are determined based on relationships between different k-nearest neighbor graphs;

[0240] Based on the dynamic graph convolutional neural network, feature extraction is performed on the first dynamic graph to obtain the first feature set.

[0241] The communication interface 701 is used to receive a first task sent by a second device, where the first task is used to request alignment of the first three-dimensional image set and the second three-dimensional image set, and the second device is the first device that executes the first task based on the weight of the first road, and the weight of the first road is associated with the data size of the three-dimensional image and the complexity of the three-dimensional image.

[0242] It should be noted that the specific processing process of the processor 702 and the communication interface 701 can be understood by referring to the above method.

[0243] Of course, in actual application, the various components in the device 700 are coupled together through the bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 704 .

[0244] The memory 703 in the embodiment of the present application is used to store various types of data to support the operation of the device 700. Examples of such data include: any computer program used to operate on the device 700.

[0245] The method disclosed in the above embodiment of the present application can be applied to the processor 702, or implemented by the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 702 or an instruction in the form of software. The above-mentioned processor 702 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 703, and the processor 702 reads the information in the memory 703 and completes the steps of the above method in combination with its hardware.

[0246] In an exemplary embodiment, the device 700 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0247] It can be understood that the memory (memory 603) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0248] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 703 storing a computer program, and the computer program can be executed by a processor 702 of a device 700 to complete the steps of the first device-side method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0249] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 702 of the device 700 to complete the steps described in the aforementioned first device-side method.

[0250] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0251] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0252] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to a first device, comprising: Determine an overlapping area between the first three-dimensional image set and the second three-dimensional image set by using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, wherein the first device is deployed in the cloud; Using the overlapping area, the first feature set and the second feature set, a third feature set is obtained, wherein the third feature set includes global features associated with the overlapping area; Performing encoding processing on the third feature set to obtain a pose matrix; The first three-dimensional image set and the second three-dimensional image set are registered using the pose matrix to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road.

2. The method according to claim 1, characterized in that The first three-dimensional image set includes M first image points, the second three-dimensional image set includes N second image points, and M and N are integers greater than or equal to 1; the use of the overlapping area, the first feature set and the second feature set to obtain a third feature set includes: Determine first features for M first image points in the first feature set, and determine second features for N second image points in the second feature set; Based on the self-attention mechanism, the first feature and the second feature are respectively updated using the overlapping area to obtain an updated first feature and an updated second feature; The third feature set is obtained by using the updated first feature and the updated second feature.

3. The method according to claim 2, characterized in that The method of updating the first feature and the second feature respectively by using the overlapping area based on the self-attention mechanism to obtain the updated first feature and the updated second feature includes: Based on the self-attention mechanism, determine a third feature and a fourth feature, wherein the third feature represents the degree of attention paid to a feature in the first feature associated with the overlapping area, and the fourth feature represents the degree of attention paid to a feature in the second feature associated with the overlapping area; Using the third feature to update the first feature to obtain an updated first feature; The second feature is updated using the fourth feature to obtain an updated second feature.

4. The method according to claim 1, characterized in that The method further comprises: determining one or more regions in the registered three-dimensional image, each region comprising one or more third image points; Performing feature extraction on the third image points corresponding to each of the one or more regions to obtain one or more fifth features; Performing fusion processing on the one or more fifth features to obtain a sixth feature; Using the sixth feature, an abnormal situation on the first road is identified.

5. The method according to claim 4, characterized in that The registered three-dimensional image includes L image layers, where L is an integer greater than or equal to 1; the step of extracting features from the third image points corresponding to each of the one or more regions to obtain one or more fifth features includes: At each image layer, feature extraction is performed on the third image points corresponding to each of the one or more regions to obtain one or more seventh features; The one or more fifth features are obtained by using the one or more seventh features corresponding to the L image layers.

6. The method according to claim 1, characterized in that The method further comprises: Based on a dynamic graph convolutional neural network, extract features from the first three-dimensional image set to obtain the first feature set; Based on a dynamic graph convolutional neural network, feature extraction is performed on the second three-dimensional image set to obtain the second feature set.

7. The method according to claim 6, characterized in that The first three-dimensional image set includes M first image points, where M is an integer greater than or equal to 1; the first three-dimensional image set is subjected to feature extraction based on the dynamic graph convolutional neural network to obtain the first feature set, including: For each first image point, using the first image point and k first image points adjacent to the first image point, obtain a k nearest neighbor graph corresponding to the first image point, where k is an integer greater than or equal to 1; Using k-nearest neighbor graphs corresponding to the M first image points, a first dynamic graph corresponding to the first three-dimensional image set is obtained, wherein edges in the first dynamic graph are determined based on relationships between different k-nearest neighbor graphs; Based on the dynamic graph convolutional neural network, feature extraction is performed on the first dynamic graph to obtain the first feature set.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: A first task is received from a second device, wherein the first task is used to request registration of the first three-dimensional image set and the second three-dimensional image set, and the second device is a first device that determines to execute the first task based on a weight of the first road, wherein the weight of the first road is associated with a data size of the three-dimensional image and a complexity of the three-dimensional image.

9. An image processing device, characterized in that: Set up on the first device, including: A first determining unit is configured to determine an overlapping area between the first three-dimensional image set and the second three-dimensional image set by using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, wherein the first device is deployed in the cloud; a second determining unit, configured to obtain a third feature set by using the overlapping area, the first feature set and the second feature set, wherein the third feature set includes global features associated with the overlapping area; An encoding unit, used for encoding the third feature set to obtain a pose matrix; A registration unit is used to register the first three-dimensional image set and the second three-dimensional image set using the posture matrix to obtain a registered three-dimensional image, and the registered three-dimensional image is used to identify abnormal conditions of the first road.

10. A device, characterized in that: include: Processor and communication interface; wherein, The processor is configured to determine an overlapping area between the first three-dimensional image set and the second three-dimensional image set by using a first feature set corresponding to the first three-dimensional image set and a second feature set corresponding to the second three-dimensional image set, wherein the device is deployed in the cloud; and obtain a third feature set by using the overlapping area, the first feature set, and the second feature set, wherein the third feature set includes global features associated with the overlapping area; The third feature set is encoded to obtain a pose matrix; and the first three-dimensional image set and the second three-dimensional image set are registered using the pose matrix to obtain a registered three-dimensional image, which is used to identify abnormal conditions of the first road.

11. A device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 8.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images