An image processing method, device, computer equipment and storage medium

Abstract

Embodiments of the present application provide an image processing method and device, computer equipment and a storage medium, wherein the method comprises: obtaining feature information of elements included in a first image and feature information of elements included in a second image, the first image comprising an environmental image of a location collected by a data collection device, the second image being an image similar to the first image in an image library included in map data; matching elements in the first image and elements in the second image according to the feature information of the elements included in the first image and the feature information of the elements included in the second image, to obtain an element matching result; and updating the image library based on the element matching result and the first image, which can improve the accuracy of updating the image library.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an image processing method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of computer technology, the map has become an important tool for users and enterprises. For users, the map can be more simple to query smaller locations, and can find the nearest and most convenient route. For enterprises, using the map can more easily push enterprise information to the user's eyes. Because the real environment may be constantly changing, the current acquired image may not be better matched with the image in the map. The prior art is mainly based on the similarity between images. The similarity between images is roughly as follows: the similarity between the current frame image and the image included in the map data is calculated, and the image matching is performed according to the similarity between the current frame image and the image included in the map data, to determine whether to update the map data; but the matching method based on the similarity between images is easily affected by light and angle changes, and does not use relative geometric position constraints, resulting in inaccurate updating of the image included in the map data. SUMMARY

[0003] The embodiments of the present application provide an image processing method and device, computer equipment and storage medium, which can improve the accuracy of updating the image library.

[0004] In one aspect, the embodiments of the present application provide an image processing method, which comprises:

[0005] Obtaining feature information of elements included in a first image and feature information of elements included in a second image, the first image comprising an environmental image of a location acquired by a data acquisition device, and the second image being an image similar to the first image in an image library included in map data;

[0006] Matching elements in the first image and elements in the second image according to the feature information of the elements included in the first image and the feature information of the elements included in the second image, to obtain an element matching result;

[0007] Updating the image library based on the element matching result and the first image.

[0008] In one aspect, the embodiments of the present application provide an image processing device, which comprises:

[0009] The acquisition unit is configured to acquire feature information of elements included in a first image and feature information of elements included in a second image, the first image including an environmental image of a location acquired by a data acquisition device, and the second image being an image similar to the first image in an image library included in map data;

[0010] The processing unit is configured to match elements in the first image and elements in the second image according to the feature information of the elements included in the first image and the feature information of the elements included in the second image, to obtain an element matching result.

[0011] The processing unit is further configured to perform update processing on the image library based on the element matching result and the first image.

[0012] In an aspect, the embodiments of the present application provide a computer device, which includes an input device, an output device, a processor and a computer storage medium, the processor and the computer storage medium are connected to each other, wherein the computer storage medium is configured to store a computer program, and the processor is configured to invoke the computer program to execute the image processing method.

[0013] In an aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image processing method.

[0014] In an aspect, the embodiments of the present application provide a computer program product, which includes a computer program; the computer program is stored in a computer readable storage medium, and the computer program is executed by a processor of a computer device to execute the business processing method.

[0015] In the embodiments of the present application, the computer device can acquire feature information of elements included in a first image and feature information of elements included in a second image, and match elements in the first image and elements in the second image according to the feature information of the elements included in the first image and the feature information of the elements included in the second image, to obtain an element matching result; and then perform update processing on an image library based on the element matching result and the first image, so as to update the image library included in map data by the element matching manner, which can reduce the problem of inaccurate update of the image library included in map data due to long-time span of images, seasonal, weather, illumination change, serious distortion, low frame rate and the like, and improve the accuracy of update of the image library included in map data. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0017] Figure 1a is an architecture diagram of an image processing system provided by an embodiment of the present application;

[0018] Figure 1b is a flowchart of an image processing scheme provided by an embodiment of the present application;

[0019] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0020] Figure 3a is a schematic diagram of a plurality of two-dimensional coordinates corresponding to a three-dimensional coordinate in the same scene provided by an embodiment of the present application;

[0021] Figure 3b is a schematic diagram of a plurality of two-dimensional coordinates corresponding to a three-dimensional coordinate in the same scene provided by an embodiment of the present application;

[0022] Figure 3c is a schematic diagram of two-dimensional coordinates in an image corresponding to a three-dimensional coordinate and two-dimensional coordinates not corresponding to a three-dimensional coordinate provided by an embodiment of the present application;

[0023] Figure 3d is a schematic diagram of matching elements in two images provided by an embodiment of the present application;

[0024] Figure 4 is a flowchart of an image processing method provided by an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of an overall architecture of image processing and a structure of element network matching provided by an embodiment of the present application;

[0026] Figure 6 is a structural schematic diagram of an image processing device provided by an embodiment of the present application;

[0027] Figure 7 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0029] In order to better realize image processing and update the image library included in the map data, the present application provides an image processing system and an image processing scheme. Please refer to Figure 1a , Figure 1a The image processing system provided by the present application can include at least one terminal device 101 and at least one server 102. The terminal device 101 can be a device dedicated to image data collection. Any device with a camera can be installed on the terminal device 101. The terminal device 101 can call the camera to collect the environmental image of the location where the terminal device 101 is located, and send the collected environmental image to the server 102, so that the server 102 updates the map data according to the collected environmental image. The terminal device 101 can also collect the environmental image of the location where the terminal device 101 is located, and update the map data based on the collected environmental image, and then save the updated map data in the server 102.

[0030] The terminal device 101 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a vehicle-mounted computer, a smart home, a wearable electronic device, a VR / AR device, etc. The terminal device 101 and the server 102 can be directly or indirectly connected through wired or wireless communication. The server 102 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc.

[0031] Based on the above-mentioned image processing system, the present application provides an image processing scheme. The image processing scheme can be as follows Figure 1bAs shown, the image processing scheme can be executed by a computer device, which can be the terminal device 101 or the server 102 described above, and the general principle of the image processing scheme is as follows: the elements of the image acquired at the current position or the current view and the elements of the images in the image library included in the map data are matched according to whether they belong to the same entity in the real environment, and the description features and geometric features corresponding to the detection results of the elements are subjected to MLP (Multi-layer Perceptron) operation and feature fusion to obtain features (hereinafter referred to as fused features) for element matching, and the fused features are subjected to element matching through an optimal transport algorithm (such as sinkhorn algorithm). Specifically, when the data acquisition device collects a current frame image (i.e., a first image), the first image includes an environment image of the current position or view collected by the data acquisition device, and then the computer device can acquire the first image and query a plurality of images similar to the first image from the image library included in the map data, obtain a three-dimensional coordinate set (or a 3D point cloud set) corresponding to each image by voting on the plurality of images, and the three-dimensional coordinate set can include one or more three-dimensional coordinates.

[0032] Then, the plurality of images are screened according to the 3D point cloud set to obtain a second image similar to the first image. Next, the computer device can generate an image pair from the first image and the second image, and perform element detection on the first image to obtain a detection result of the first image, and perform element detection on the second image to obtain a detection result of the second image, wherein each detection result actually corresponds to feature information of an element; then the feature information of the elements included in each image in the image pair is input into an element matching network for feature fusion to obtain first fused features of the elements included in the first image and second fused features of the elements included in the second image. The feature information can include geometric features and description features (also referred to as visual features or descriptor features) of the elements, and the description features are used to describe the category of the element, the outline of the element, etc. Then, the computer device performs inner product calculation on the first fused features of the elements in the first image and the second fused features of the elements in the second image to obtain an element association score between the elements included in the first image and the elements included in the second image, and the element association score between the elements included in the first image and the elements included in the second image is stored in a matching matrix, and the matching matrix is calculated through a Sinkhorn algorithm (an optimal transport algorithm). Then, the elements in the first image and the elements in the second image are matched according to the calculated matching matrix. After obtaining the element matching result, the computer device can update the image library based on the obtained element matching result and the first image.

[0033] The image processing scheme can be applied to a map data automatic updating scene. In the map data automatic updating scene, the image processing scheme provided in the application can achieve an accuracy rate of 98% and a recall rate of more than 90% in element association matching in a high-difficulty task of full-element matching of map crowd-sourced data, reduces the problem of inaccurate map data updating due to factors such as long-time span of images, seasonal changes, weather changes, light changes, serious distortion, and low frame rate, improves the accuracy of updating of an image library included in map data, and improves the automation degree of a map data production process. Moreover, the element matching does not need any human-designed strategy for matching the first image and the images in the database, can effectively solve the ambiguity problem of matching, and saves a large amount of manpower.

[0034] Please refer to Figure 2 , Figure 2 A flowchart of an image processing method provided in an embodiment of the application is shown. The image processing method can be executed by a computer device, which can be the terminal device 101 or the server 102. The image processing method can include the following steps S201-S203.

[0035] S201, obtaining feature information of elements included in a first image and feature information of elements included in a second image. The first image includes an environmental image of a location collected by a data collection device, and the second image is an image similar to the first image in an image library included in map data.

[0036] The first image can include one or more elements. The element generally refers to an element containing semantic information, and the element includes but is not limited to a traffic signal lamp, a signboard, an electronic police, a zebra crossing, a ground arrow, a trademark plate, and the like in map data. The feature information can include a description feature and a geometric feature of the element. The description feature can be a category and a confidence of the element, and the like. The element category can include but is not limited to the traffic signal lamp, the signboard, the electronic police, and the like. The data collection device can be the terminal device 101 or other devices specially used for data collection, which is not limited in the embodiment of the application. The number of the second images can be one or more. The candidate images in the image library can include one or more elements. Similarly, the second image can include one or more elements. The first image and the second image can be understood as follows: the first image and the second image belong to images in the same environment (or the same scene), and the first image and the second image can be collected by using the same data collection device at different angles, or the first image and the second image can be collected by using different data collection devices at the same angle. For example, the first image and the second image both belong to images in a scene corresponding to a crossroad in XX street.

[0037] In a specific implementation, the data collection device is configured with a camera component or an image collection sensor capable of collecting images; when a user wants to collect an environmental image of a certain location or a certain perspective, the user here can be an ordinary user using map data; or the user here can also be a map management user who wants to update the map data, etc.; the data collection device can be used to collect an environmental image of the location or the perspective. Then the data collection device can call the collection of a first image using the camera component or the image collection sensor of the image, which includes the environmental image of the location or the perspective collected by the data collection device; then the computer device can obtain the first image, and obtain a second image similar to the first image from the image library included in the map data; then the first image is detected for elements to obtain the feature information of the elements included in the first image, and the second image is detected for elements to obtain the feature information of the elements included in the second image. It should be understood that when the computer device is the data collection device, the computer device can directly obtain the first image; when the computer device is not the data collection device, the computer device needs to obtain the first image from the data collection device.

[0038] In one embodiment, the computer device detects elements in the first image to obtain the feature information of the elements included in the first image, and detects elements in the second image to obtain the feature information of the elements included in the second image, which is implemented in the following manner: the computer device can use a target detection box to detect elements in the first image to obtain the feature information of the elements included in the first image, and use a target detection box to detect elements in the second image to obtain the feature information of the elements included in the first image; wherein the target detection box can be a two-dimensional detection box (or 2D detection box), or the target detection box can also be other detection boxes for element detection. The above-mentioned feature information can include description features (description features can be referred to as descriptor features) and geometric features; wherein the description features are used to describe the element category, confidence, etc. of the element; the geometric features can include the center point coordinates, length, width, etc. of the target detection box.

[0039] In one embodiment, the first image includes an environmental image of a location collected by the data collection device in real time, for example, when the data collection device is at location 1, the first image collected by the data collection device includes an environmental image of the location 1 where the data collection device is located; when the data collection device is at location 2, the first image collected by the data collection device includes an environmental image of the location 2 where the data collection device is located. In another embodiment, the first image is an environmental image of a target location collected by the data collection device within a preset period. For example, the preset period is 2 days; the target location is location 1; the first image can be an environmental image of the location 1 collected by the data collection device with a period of two days. It can be understood that the data collection device will collect the first image at location 1 every two days.

[0040] In one embodiment, the map data includes an image library including a plurality of candidate images; the specific implementation of the computer device obtaining a second image similar to the first image from the image library included in the map data can include the following two ways:

[0041] (1) Calculate the similarity between the first image and each of the plurality of candidate images, and select the second image similar to the first image according to the similarity.

[0042] In a specific implementation, the computer device can calculate the similarity between each candidate image and the first image, and then determine the second image similar to the first image from the image library according to the similarity between each candidate image and the first image. For example, the image library includes 3 candidate images, which are candidate image 1, candidate image 2 and candidate image 3; the computer device can calculate the similarity between candidate image 1 and the first image, the similarity between candidate image 2 and the first image, and the similarity between candidate image 3 and the first image.

[0043] As an implementation, the specific implementation of determining the second image similar to the first image from the image library according to the similarity between each candidate image and the first image is that the computer device can determine the candidate image with a similarity greater than or equal to a similarity threshold as the second image similar to the first image. Wherein, the similarity threshold can be set according to requirements; for example, the similarity between candidate image 1 and the first image is greater than the similarity threshold, the similarity between candidate image 2 and the first image is less than the similarity threshold, and the computer device determines candidate image 1 with a similarity greater than the similarity threshold as the second image similar to the first image.

[0044] As another implementation, the computer device can arrange the candidate images included in the image library in descending order according to the similarity between each candidate image and the first image, and then select the top K candidate images as the second image similar to the first image. Wherein, K is an integer greater than or equal to 1; K can be set according to requirements, for example, the value of K is 1. The computer device arranges the three candidate images in descending order according to the similarity between candidate image 1 and the first image, the similarity between candidate image 2 and the first image, and the similarity between candidate image 3 and the first image, and sets the three candidate images arranged in descending order as: candidate image 1, candidate image 3 and candidate image 2; the computer device selects the top candidate image 1 from the three candidate images as the second image similar to the first image; that is, candidate image 1 is taken as the second image similar to the first image.

[0045] (2) calling a retrieval network to obtain one or more images similar to the first image from a candidate image included in an image library; and then determining, by the computer device, a second image from the one or more images according to three-dimensional coordinates corresponding to each image and two-dimensional coordinates corresponding to each three-dimensional coordinate.

[0046] In the implementation process, the computer device calls a retrieval network to obtain one or more images similar to the first image from an image library included in the map data, where the retrieval network can be a neural network model for screening images, and the retrieval network can be a retrieval network. Each candidate image in the image library corresponds to one or more three-dimensional coordinates. It can be understood that the one or more three-dimensional coordinates corresponding to each candidate image actually indicate one or more elements included in each candidate image; that is, if each image corresponds to three three-dimensional coordinates, each image includes three elements. Each three-dimensional coordinate corresponds to one or more two-dimensional coordinates. As shown in Figure 3a ; Figure 3a For visualization of the three-dimensional coordinates of the sign corresponding to the two-dimensional coordinates in the three images in the same scene; it can be seen that the three-dimensional coordinates of the sign in the same scene can correspond to three two-dimensional coordinates (i.e. Figure 3a 31 indicates the two-dimensional coordinates in FIG. 1), where the above three images can be imaging data of the same scene at different positions or angles. For example, the three images can be images captured by the same camera at different times, or the multiple images are images captured by different cameras at different times, or the multiple images are images captured by different cameras at the same time. Similarly in Figure 3b , the three-dimensional coordinates of the traffic light can also correspond to three two-dimensional coordinates.

[0047] In one embodiment, the one or more three-dimensional coordinates corresponding to each candidate image can be obtained by the computer device through a voting algorithm to collect one or more three-dimensional coordinates (or 3D points) of each candidate image. The one or more three-dimensional coordinates corresponding to each candidate image can form a three-dimensional coordinate set (also referred to as a 3D point cloud set), and the three-dimensional coordinate set (also referred to as the 3D point cloud set) corresponding to each candidate image is represented as:

[0048]

[0049] where P i represents the three-dimensional coordinate set of the i-th candidate image, R 3×N represents three-dimensional; N represents the number of three-dimensional coordinates. Each three-dimensional coordinate can correspond to one or more two-dimensional coordinates. Each candidate image can include one or more elements.

[0050] It can be understood that each of the one or more images similar to the first image obtained by searching the network also corresponds to one or more three-dimensional coordinates, and each three-dimensional coordinate also corresponds to one or more two-dimensional coordinates (or 2D points). Then, the computer device can determine the second image from the one or more images according to the three-dimensional coordinates corresponding to each image and the two-dimensional coordinates corresponding to each three-dimensional coordinate.

[0051] In one embodiment, since the multiple images obtained by searching the network may have outliers (i.e., the images include inconsistent environmental images, which can also be understood as inconsistent scenes. For example, the multiple images correspond to two scenes, the first scene is a crossroad, and the second scene is a three-way intersection), a consensus calculation can be performed on the multiple images similar to the first image obtained by searching the network to filter out images with inconsistent scenes, thereby reducing the amount of calculation for subsequent element matching. In this case, the specific implementation of the computer device for determining the second image from the one or more images according to the three-dimensional coordinates corresponding to each image and the two-dimensional coordinates corresponding to each three-dimensional coordinate is that the computer device obtains target three-dimensional coordinates corresponding to a number of two-dimensional coordinates that exceeds a number threshold from the one or more three-dimensional coordinates corresponding to each image; it can also be understood that the computer device filters out three-dimensional coordinates corresponding to a number of two-dimensional coordinates that does not exceed the number threshold from the one or more three-dimensional coordinates corresponding to each image. The number threshold can be set according to requirements, for example, the number threshold can be set to 3, 5, etc. Then the computer device can obtain a first total number of two-dimensional coordinates corresponding to each image and a second total number of two-dimensional coordinates corresponding to the target three-dimensional coordinates; then determine the reconstruction rate of each image according to the first total number and the second total number; then the computer device can determine the image with a reconstruction rate greater than or equal to a target threshold from the one or more images, and take the image with a corresponding reconstruction rate greater than or equal to the target threshold from the one or more images as the second image. The target threshold can be set according to requirements. The reconstruction rate of each image actually refers to the reconstruction rate of the three-dimensional coordinates in each image, which is used to indicate the probability of reconstructing a two-dimensional image into a three-dimensional image.

[0052] For example, the target threshold is 10%; as Figure 3c shown, Figure 3c the left image in the middle is the collection trajectory of the three-dimensional coordinates corresponding to the target image collected by using the voting algorithm, and the three-dimensional coordinates in the multiple 3D point cloud sets, and Figure 3c the right image in the middle represents the two-dimensional coordinates in the target image, and among the two-dimensional coordinates in the right image, there are 4595 two-dimensional coordinates (i.e. Figure 3c black points in the middle) without corresponding three-dimensional coordinates; there are 573 two-dimensional coordinates with corresponding three-dimensional coordinates (i.e. Figure 3cThe target image corresponds to a three-dimensional coordinate reconstruction rate of 573 / (4595+573)=11.09%. Then the computer device can determine that the three-dimensional coordinate reconstruction rate corresponding to the target image is greater than 10%, and the target image is taken as the second image similar to the first image.

[0053] In an embodiment, the above-mentioned map data includes an image library which can be obtained by collecting environment images by a target sensor and processing the collected environment images. The target sensor can include one or more of the following: an image sensor (i.e., a camera), a GPS (Global Positioning System) sensor, and an IMU (Inertial Measurement Unit) sensor. For example, the environment images collected by the target sensor have noise, and the environment image data needs to be de-noised. Then, the environment images after de-noising are stored in the image library. The image library includes a plurality of candidate images which can be images collected at different times by the same camera, images collected at different times by different cameras, and images collected at the same time by different cameras, and the like.

[0054] S202, according to the feature information of the elements included in the first image and the feature information of the elements included in the second image, matching the elements in the first image and the elements in the second image to obtain an element matching result.

[0055] The element matching result can indicate whether there is a matching element between the elements included in the first image and the elements included in the second image. The so-called matching element refers to two elements belonging to the same physical point. The physical point refers to the same entity belonging to the same spatial position in the real environment. In an embodiment, the element matching result can include a matching element. The matching element can be an element in the first image and an element in the second image. For example, the matching element is element A in the first image and element B in the second image, i.e., element A and element B are matching elements. Or, the element matching result can include elements that are not matched between the elements in the first image and the elements in the second image. For example, the elements that are not matched include element A in the first image and element B in the second image, i.e., element A and element B are not matched elements.

[0056] In an embodiment, the specific implementation of step S202 can be that the computer device can directly compare the feature information of the elements included in the first image with the feature information of the elements included in the second image in pairs, determine whether there is a matching element between the elements included in the first image and the elements included in the second image according to the comparison result, and obtain an element matching result. For example, the feature information of element 1 included in the first image is compared with the feature information of element 2 included in the second image, the feature information of element 1 is the same as the feature information of element 2, and the computer device determines that element 1 and element 2 are matching elements. For another example, the feature information of element 3 included in the first image is compared with the feature information of element 4 included in the second image, the feature information of element 3 is not the same as the feature information of element 4, and the computer device determines that element 3 and element 4 are not matching elements, that is, the element matching result includes non-matching elements.

[0057] In an embodiment, since it is tedious to directly compare the feature information of the elements included in the first image with the feature information of the elements included in the second image, and it also leads to inaccurate element matching result, the specific implementation of step S202 can also be that the computer device can further call the element matching network to perform feature fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image, obtain first fusion features of the elements included in the first image and second fusion features of the elements included in the second image, and perform matching on the elements in the first image and the elements in the second image according to the first fusion features and the second fusion features, to obtain the element matching result. The first fusion features and the second fusion features can be understood as features constructed for element matching.

[0058] S203, updating the image library based on the element matching result and the first image.

[0059] As known from step S202, the element matching result is used to indicate whether there is a matching element between the elements included in the first image and the elements included in the second image.

[0060] In one embodiment, if the element matching result indicates that the target element exists in the elements included in the first image, the target element is any element in the first image, and the second image does not include the target element, it indicates that the target element in the first image is not found in the second image. The element matching result indicates that the target element is a new element, and the computer device updates the image library using the first image. Specifically, the computer device can replace the second image similar to the first image in the image library with the first image. Or the computer device can replace the elements included in the second image in the image library with the elements included in the first image. In another embodiment, if the element matching result indicates that there is a matching element, the matching element is a first element included in the first image and a second element included in the second image, wherein the matching means that the first element and the second element correspond to the same entity; the computer device can establish an association relationship between the first element and the second element; for example, Figure 3d As shown in Figure 3d , the image 1 and the image 2 in the image library belong to the same scene; in Figure 3d , four pairs of matching elements are exemplarily listed; they are the element 101 in the image 1 and the element 101 in the image 2; the element 102 in the image 1 and the element 102 in the image 2, the element 103 in the image 1 and the element 103 in the image 2; the element 104 in the image 1 and the element 104 in the image 2; the computer device can establish an association relationship between each pair of matching elements in the four pairs of matching elements. Then, after establishing the association relationship between the first element in the first image and the second element in the second image, the computer device updates the image library according to the association relationship and the first image. Specifically, the computer device does not update the second element of the second image in the image library according to the association relationship between the first element in the first image and the second element in the second image, and then updates the second image in the image library using other elements included in the first image, so that when updating the image library, it is not necessary to match the elements one by one according to the matching strategy, but only to determine the matching elements according to the association relationship, and to update the elements in the second image in the image library which do not match the elements in the first image according to the first image, thereby improving the automatic processing capability of the map data.

[0061] In one embodiment, if the element matching result indicates that the elements included in the first image and the elements included in the second image are completely matched, for example, the first image includes three elements, the second image includes three elements, and the three elements included in the first image and the three elements included in the second image are completely matched two by two, the computer device does not need to update the image library.

[0062] In this embodiment, the computer device can acquire feature information of elements included in the first image and feature information of elements included in the second image, and match the elements in the first image and the elements in the second image based on the feature information of elements included in the first image and the feature information of elements included in the second image to obtain feature matching results; then, based on the feature matching results and the first image, the image library is updated. By updating the image library included in the map data through feature matching, the problem of inaccurate updates to the image library included in the map data due to seasonal, weather, lighting changes, severe distortion, low frame rate, etc., over long periods of time can be reduced, thereby improving the accuracy of updating the image library included in the map data.

[0063] Please see Figure 4 , Figure 4 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The image processing method can be executed by a computer device, which can be the terminal device 101 or the server 102 described above; the image processing method may include the following steps S401-S404:

[0064] S401. Obtain feature information of the elements included in the first image and feature information of the elements included in the second image. The first image includes an environmental image of the location collected by the data acquisition device, and the second image is an image similar to the first image in the image library included in the map data.

[0065] In a specific implementation, the computer device can use a target detection box to perform feature detection on the first image, obtaining a detection result including the feature information of the features in the first image. Then, the target detection box can be used to perform feature detection on the second image, obtaining a detection result including the feature information of the features in the second image. For example, Figure 5 For the overall architecture of image processing and the structure of feature network matching; in Figure 5 In the image, the first image is image I. A The second image is image I. B Computer equipment uses target detection boxes to analyze image I. A By performing feature detection, we can obtain results such as Figure 5 Image I shown A It includes four detection results, each corresponding to image I. A The feature information of one element. Similarly, computer devices use target detection boxes to analyze image I. B By performing feature detection, we can obtain results such as Figure 5 Image I shown B It includes 5 detection results, each corresponding to image I. Bcharacteristic information of one element in the image; wherein the target detection frame can be a two-dimensional detection frame (or referred to as a 2D detection frame), or the target detection frame can also be other detection frames for element detection. Wherein the above characteristic information can include description characteristics (the description characteristics can be referred to as description sub-characteristics) and geometric characteristics; wherein the description characteristics are used to describe the element category, confidence and the like of the element; the geometric characteristics can be obtained by calling the MLP to fuse a plurality of geometric sub-characteristics, so as to realize dimensionality reduction processing on the geometric characteristics. The geometric sub-characteristics can include one or more of the following: the center point coordinates of the target detection frame, the length, the width and the like, and the geometric characteristics can be represented as:

[0066] fp = MLP (|x, y, w, h, s|)

[0067] Wherein, f p represents the geometric characteristics, MLP() represents dimensionality reduction processing; x and y represent the center point coordinates of the target detection frame; w and h respectively represent the width and height of the target detection frame; s represents the confidence.

[0068] It should be noted that the geometric characteristics of the elements included in any image can be obtained by calling the MLP to fuse the geometric sub-characteristics.

[0069] In one embodiment, the number of second images can be multiple, and the computer device can generate a plurality of image pairs by taking the first image and the plurality of second images. For example, the plurality of image pairs can be represented as: {(I q ,I db0 ),(I q ,I db1 ),(I q ,I db2 )...}, wherein I q represents the first image, I db0 , I db1 , I db2 represent the plurality of second images. The step S401 can obtain the characteristic information of the first image and the characteristic information of the second image in each image pair. Wherein the obtaining of the characteristic information of the first image and the characteristic information of the second image in each image pair can refer to the implementation manner of the embodiments provided in the above Figure 2 The provided embodiments correspond to the implementation manner, which will not be described here.

[0070] S402, calling an element matching network to fuse the characteristic information of the elements included in the first image and the characteristic information of the elements included in the second image, to obtain first fusion characteristics of the elements included in the first image and second fusion characteristics of the elements included in the second image.

[0071] Wherein, the element matching network can be as Figure 5As shown, the element matching network includes a feature fusion layer and an optimal matching layer. The feature fusion layer can construct features for element matching. In order to ensure the accuracy of element matching, the feature fusion layer needs to fuse the features of elements within a graph and cross-fuse the features of elements between graphs. Therefore, the feature fusion layer can include a self-interaction layer (self layer) and a cross layer (cross layer). The self-interaction layer is used for feature fusion of elements within a graph, and the cross layer is used for feature fusion of elements between graphs. The optimal matching layer is used for element matching of elements included in two images.

[0072] In one embodiment, the specific implementation of step S402 is that the computer device calls the feature fusion layer of the element matching network to respectively perform intra-image fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image, to obtain the internal fusion features of the elements included in the first image and the internal fusion features of the elements included in the second image. Specifically, the computer device calls the self-interaction layer in the feature fusion layer to perform intra-image fusion on the feature information of the elements included in the first image, to obtain the internal fusion features of the elements included in the first image, and calls the self-interaction layer in the feature fusion layer to perform intra-image fusion on the feature information of the elements included in the second image, to obtain the internal fusion features of the elements included in the second image. For example, in the embodiment shown in FIG. 2, the first image is image I Figure 5 A The input of the feature fusion layer of the element matching network is the image I A The geometric features of the elements included in the image I and the description features It can be understood that the image I A The geometric features of the i-th element included in the image I and the description features Suppose that the number of elements included in the first image is 4, which are element 1, element 2, element 3, and element 4. Then the computer device calls the self-interaction layer in the feature fusion layer to perform self-interaction processing on the geometric features and the description features of the element 1 with the geometric features and the description features of the element 2, the geometric features and the description features of the element 3, and the geometric features and the description features of the element 4, to obtain the internal fusion features of the element 1. Similarly, the computer device can calculate the internal fusion features of the element 2, the internal fusion features of the element 3, and the internal fusion features of the element 4. Similarly, for the image I B The input of the feature fusion layer of the element matching network is the image I B The geometric features of the elements included in the image I and the description features It can be understood that the image I B The geometric features of the i-th element included in the image I and the description features Then the internal fusion features of the elements included in the image I A ​The implementation of the internal fusion feature of the included element can obtain the image I B The internal fusion feature of the included element.

[0073] Then, the computer device calls the feature fusion layer of the element matching network to perform image inter-fusion on the feature information of the element included in the first image and the feature information of the element included in the second image, to obtain the cross-fusion feature of the element included in the first image and the cross-fusion feature of the element included in the second image; for example, the image I B includes three elements, which are element 5, element 6 and element 7 respectively; for the image I A , the computer device calls the cross layer in the feature fusion layer to cross-process the feature information of element 1 and element 5, cross-process the feature information of element 1 and the feature information of element 6, and cross-process the feature information of element 1 and the feature information of element 7, to finally obtain the cross-fusion feature of element 1. Similarly, the computer device can determine the cross-fusion features of the remaining elements included in the image I A . B The cross-fusion feature of the element included in the image I A .

[0074] After obtaining the internal fusion feature and the cross-fusion feature of the element included in the first image, the computer device determines the first fusion feature of the element included in the first image according to the internal fusion feature and the cross-fusion feature of the element included in the first image. In a specific implementation, the computer device iterates the internal fusion feature and the cross-fusion feature of the element included in the first image for N times to obtain the first fusion feature of the element included in the first image. The first fusion feature can be used for element matching. The first fusion feature of the element included in the first image can be represented as f A A A A B A A A B The dimension of the first fusion feature f A of the element included in the first image is: f A ∈R n1×d ​​​​​​​​; n1 represents the number of detection results corresponding to the first image, which can also refer to the number of elements included in the first image, and d represents the dimension, such as 256 dimensions. The first fusion feature of the i-th element included in the first image can be represented as

[0075] After obtaining the internal fusion feature and the cross fusion feature of the element included in the second image, the second fusion feature of the element included in the second image is determined according to the internal fusion feature and the cross fusion feature of the element included in the second image. In a specific implementation, the computer device iterates the internal fusion feature and the cross fusion feature of the element included in the second image for N times to obtain the second fusion feature of the element included in the second image. The second fusion feature is used for element matching. The second fusion feature of the element included in the second image can be represented as f B , that is, f B = [self(f B ,f B ), cross(f B ,f A )]xN, self(f B ,f B ) can be understood as the intra-image fusion of the element included in the image B with the element included in the image B; cross(f B ,f A ) can be understood as the inter-image fusion of the element included in the image B with the element included in the image A. Wherein, the dimension of the second fusion feature f A of the element included in the second image is: f B ∈R n2×d , n2 is the number of detection results corresponding to the second image, which can also refer to the number of elements included in the second image, and d represents the dimension. The second fusion feature of the i-th element included in the second image can be represented as

[0076] In an embodiment, before performing step S402, the initial neural network needs to be trained to obtain the element matching network. The computer device can obtain a training sample set, the training sample set including positive sample pairs, negative sample pairs, and label information of each sample pair, the label information including feature information of elements in each sample of each sample pair; then calling the initial neural network to process the positive sample pairs and the negative sample pairs to obtain predicted matching elements corresponding to each sample pair; and adjusting the network parameters of the initial neural network by using the label information of each sample pair and the corresponding predicted matching elements to obtain the element matching network. Wherein, the samples included in the positive sample pairs have elements corresponding to the same entity, and the positive sample pairs can be image pairs obtained in the same scene (or the same environment), for example, the positive sample pairs include two images obtained in a crossroad scene, which are image 1 and image 2. The positive sample pairs include elements corresponding to the same entity in image 1 and image 2. For example, the positive sample pairs include an element indicator in image 1 and an element indicator in image 2, and the element indicator in image 1 and the element indicator in image 2 correspond to the same entity. The samples included in the negative sample pairs do not have elements corresponding to the same entity. The negative sample pairs can be image pairs in different scenes, or the negative sample pairs can also be image pairs obtained in the same scene. Since the image pairs obtained in the same scene can have elements corresponding to the same entity, or can not have elements corresponding to the same entity; therefore, the elements corresponding to the same entity in the image pairs can be regarded as positive sample pairs, and the label information of the above positive sample pairs is the feature information of the elements corresponding to the same entity. In the elements included in image 1 and the elements included in image 2, there are elements that do not correspond to the same entity; then the elements in the image pairs that do not correspond to the same entity can be regarded as negative sample pairs, at this time, the label information of the negative sample pairs is the feature information of the elements that do not correspond to the same entity, for example, the label information of the negative sample pairs includes the feature information of the element indicator in image 1 and the feature information of the element traffic light in image 2. By obtaining image pairs in the same scene, the elements corresponding to the same entity in the image pairs are regarded as positive sample pairs, and the elements in the image pairs that do not correspond to the same entity are regarded as negative sample pairs, which can reduce the acquisition of a large amount of training data.

[0077] The specific implementation of the computer device calling the initial neural network to process positive and negative sample pairs to obtain the predicted matching elements corresponding to each sample pair can be as follows: the computer device calls the initial neural network to process positive sample pairs, that is, the initial neural network first learns positive sample pairs; then the computer device calls the initial neural network to process negative sample pairs, that is, the initial neural network learns positive sample pairs again; this is conducive to obtaining a better element matching network. The number of positive and negative sample pairs needs to meet preset conditions to achieve a balance between positive and negative sample pairs. For example, the number of negative sample pairs should not exceed 20% of the number of positive sample pairs. The computer device calls the initial neural network to process 10 positive sample pairs, that is, the initial neural network first learns these 10 positive sample pairs; then the computer device calls the initial neural network to process 2 negative sample pairs, that is, the initial neural network learns these 2 positive sample pairs again.

[0078] In one embodiment, the computer device adjusts the network parameters of the initial neural network using the annotation information of each sample pair and the corresponding predicted matching elements to obtain the element matching network. The specific implementation is as follows: the computer device calculates the model loss of the initial neural network based on the annotation information of each sample pair and the corresponding predicted matching elements, and then adjusts the network parameters of the initial neural network in the direction of decreasing model loss to obtain the element matching network. The formula for calculating the model loss is:

[0079]

[0080] Where (i,j)∈M represents the annotation information of the elements included in each sample in the positive sample pair; i∈I represents the annotation information of the elements included in sample I in the positive sample pair; j∈J represents the annotation information of the elements included in sample J in the positive sample pair.

[0081] S403. Match the elements in the first image with the elements in the second image based on the first fusion feature and the second fusion feature to obtain the element matching result.

[0082] In a practical implementation, the computer device can invoke the optimal allocation layer of the feature matching network to perform feature cross-multiplication (also known as inner product calculation) on the first fusion feature of the features in the first image and the second fusion feature of the features in the second image, to obtain the matching matrix between the features in the first image and the features in the second image. For example, image I A The first fusion feature of the i-th element is Image I B The second fusion feature of the j-th element is Computer devices can invoke the optimal allocation layer of the feature matching network. as well as Perform feature cross-multiplication to obtain the multiplication fraction S. i，j According to the product of fractions S i，j Final generated image I A Elements and Image I B The matching matrix between elements in the image. The dimension of the matching matrix can be determined by the number of elements in the first image and the number of elements in the second image. For example, in the image I above... A The first fusion feature f of the elements in A The dimension of f is: A ∈R n1×d Image I B The second fusion feature f of the elements B The dimension of f is: B ∈R n2×d The computer device invokes the optimal allocation layer of the feature matching network for image I. A The first fusion feature of the elements in the image I B The second fusion feature of the elements in the data is multiplied by feature cross to obtain I. A The elements in and I B The matching matrix M between the elements in the matrix has dimension M∈R. (n1+1)×(n2+1) In this application embodiment, consideration is given to image I A Included elements and images I B Among the included elements, there are elements that do not have a matching element. Therefore, the embodiments of this application can expand the dimension of the matching matrix, that is, the matching matrix M∈R (n1+1)×(n2+1) The +1 is used to handle situations where there are no matching elements.

[0083] After obtaining the matching matrix, an optimal transfer algorithm can be used to calculate the optimal transfer of the matching matrix, resulting in the optimal allocation matrix P between the features in the first image and the features in the second image. c The optimal allocation matrix P is used to determine the feature matching results between the features in the first image and the features in the second image. Each row of this optimal allocation matrix can represent a feature in the first image, and each column can represent a feature in the second image; or each row of this optimal allocation matrix can represent a feature in the second image, and each column can represent a feature in the first image. c The sum of the scores in each row is 1, and the sum of the scores in each column is also 1. The optimal transfer algorithm can be the sinkhorn algorithm; the scores in this optimal allocation matrix can be used to indicate the correlation between features in the first image and corresponding features in the second image.

[0084] In an embodiment, the computer device determines the specific implementation of the element matching result of the elements in the first image and the elements in the second image according to the optimal allocation matrix as follows: the computer device determines the elements in the first image and the corresponding elements in the second image in the optimal allocation matrix that meet the element matching condition as matching elements, and determines the elements in the first image and the corresponding elements in the second image that do not meet the element matching condition as non-matching elements. That is, the element matching result can include at least one of the matching elements and the non-matching elements. Wherein, since the higher the score in the optimal allocation matrix corresponds to the greater relevance of the two elements, and the elements cannot have a one-to-many association, the element matching condition can include one or more of the following: the score corresponding to the elements in the first image and the elements in the second image in the optimal allocation matrix is greater than a target threshold, for example, the target threshold is 0.2; and the score corresponding to the elements in the first image and the elements in the second image is the largest in the row and the column of the optimal allocation matrix.

[0085] S404, updating the image library based on the element matching result and the first image.

[0086] In an embodiment, the element matching result includes matching elements, which are the first elements in the first image and the second elements in the second image, and then the computer device establishes an association relationship for the matching elements. For example, in Figure 5 , after step S403, there are 4 pairs of matching elements in the obtained element matching result, and the computer device establishes an association relationship between each pair of matching elements (such as Figure 5 in the dashed line represents the association relationship established between each pair of matching elements); then the computer device can not update the matching elements in the image library according to the association relationship of the matching elements; and update the image library according to the remaining elements in the first image other than the matching elements.

[0087] In the embodiments of the present application, the computer device obtains the feature information of the first image and the feature information of the second image, and then calls the element matching network to perform feature fusion on the feature information of the first image and the feature information of the second image to obtain the first fusion feature of the elements included in the first image and the second fusion feature of the elements included in the second image; then the computer device matches the elements in the first image with the elements in the second image according to the first fusion feature and the second fusion feature to obtain the element matching result. The features obtained by the element matching network for matching can better adapt to long-time changes, changes in perspective, lens distortion, etc., and the element matching is completed by the element matching network, without the need for any human-designed strategy to obtain one-to-one matching, which can effectively solve the ambiguity problem of element matching, and is very suitable for multi-view element matching under long time and different perspectives. Then, the computer device updates the image library based on the element matching result and the first image, which can improve the accuracy of updating the image library.

[0088] Based on the image processing method provided above, a structural schematic diagram of an image processing device is provided in the embodiments of the present application. As shown in the figure, Figure 6 the image processing device can be applied to the computer device in the corresponding embodiments described above. Figure 2 or Figure 4 Specifically, the image processing device can be a computer program (including program code) running in the computer device, for example, the image processing device is an application software; the image processing device can be used to execute the corresponding steps in the method provided in the embodiments of the present application.

[0089] The acquisition unit 601 is configured to acquire feature information of elements included in a first image and feature information of elements included in a second image, the first image including an environmental image of a location collected by a data collection device, and the second image being an image similar to the first image in an image library included in map data;

[0090] The processing unit 602 is configured to match elements in the first image and elements in the second image according to the feature information of the elements included in the first image and the feature information of the elements included in the second image, to obtain an element matching result.

[0091] The processing unit 602 is further configured to perform update processing on the image library based on the element matching result and the first image.

[0092] In one embodiment, the processing unit 602 is further configured to:

[0093] Call a search network to acquire one or more images similar to the first image from candidate images included in the image library, each candidate image in the image library corresponding to one or more three-dimensional coordinates, and each three-dimensional coordinate corresponding to one or more two-dimensional coordinates.

[0094] Determine a second image from the one or more images according to the three-dimensional coordinates corresponding to each image and the two-dimensional coordinates corresponding to each three-dimensional coordinate.

[0095] In one embodiment, when the processing unit 602 determines the second image from the one or more images according to the three-dimensional coordinates corresponding to each image and the two-dimensional coordinates corresponding to each three-dimensional coordinate, the processing unit 602 can be specifically configured to:

[0096] From the one or more three-dimensional coordinates corresponding to each image, acquire a target three-dimensional coordinate whose number of corresponding two-dimensional coordinates exceeds a number threshold;

[0097] Acquire a first total number of two-dimensional coordinates corresponding to each image and a second total number of two-dimensional coordinates corresponding to the target three-dimensional coordinate.

[0098] determining a reconstruction rate of each of the one or more images according to the first total number and the second total number;

[0099] taking, as a second image, an image of the one or more images for which a corresponding reconstruction rate is greater than or equal to a target threshold.

[0100] In an embodiment, the processing unit 602, when updating the map based on the element matching result and the first image, can be specifically configured to:

[0101] if the element matching result indicates that there is a target element in the elements included in the first image, updating the image library using the first image, the target element being any element in the first image and the second image not including the target element.

[0102] In an embodiment, the processing unit 602, when matching the elements in the first image and the elements in the second image according to the feature information of the elements included in the first image and the feature information of the elements included in the second image, can be specifically configured to:

[0103] calling an element matching network to perform feature fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image to obtain first fusion features of the elements included in the first image and second fusion features of the elements included in the second image;

[0104] matching the elements in the first image and the elements in the second image according to the first fusion features and the second fusion features to obtain an element matching result.

[0105] In an embodiment, the processing unit 602, when calling an element matching network to perform feature fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image to obtain first fusion features of the elements included in the first image and second fusion features of the elements included in the second image, can be specifically configured to:

[0106] calling a feature fusion layer of the element matching network to perform intra-image fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image respectively to obtain internal fusion features of the elements included in the first image and internal fusion features of the elements included in the second image;

[0107] fusing, by calling a feature fusion layer of the element matching network, the feature information of the element included in the first image and the feature information of the element included in the second image, to obtain cross-fusion features of the element included in the first image and cross-fusion features of the element included in the second image;

[0108] determining first fusion features of the element included in the first image according to the internal fusion features and the cross-fusion features of the element included in the first image;

[0109] determining second fusion features of the element included in the second image according to the internal fusion features and the cross-fusion features of the element included in the second image.

[0110] In an embodiment, the obtaining unit 601 is further configured to obtain a training sample set, the training sample set including a positive sample pair, a negative sample pair, and label information of each sample pair, the label information including feature information of an element in each sample of the each sample pair, the positive sample pair including samples in which elements corresponding to a same entity exist, and the negative sample pair including samples in which elements corresponding to a same entity do not exist;

[0111] The processing unit 602 is further configured to process the positive sample pair and the negative sample pair by calling an initial neural network to obtain predicted matching elements corresponding to the each sample pair; and adjust network parameters of the initial neural network by using the label information of the each sample pair and the predicted matching elements corresponding to the each sample pair to obtain an element matching network.

[0112] It can be understood that the functions of the functional units of the image processing apparatus in this embodiment can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the related descriptions of the above method embodiments Figure 2 or Figure 4 , which will not be described here in detail.

[0113] In the embodiments of the present application, the computer device can obtain feature information of an element included in a first image and feature information of an element included in a second image, and match the element in the first image and the element in the second image according to the feature information of the element included in the first image and the feature information of the element included in the second image to obtain an element matching result; and then update an image library based on the element matching result and the first image. The image library included in the map data is updated by the element matching manner, which can reduce the problem of inaccurate updating of the image library included in the map data due to long-time span of images, such as seasonal, weather, illumination change, serious distortion, low frame rate, etc., and improve the accuracy of updating the image library included in the map data.

[0114] Further, please refer to Figure 7 ,Figure 7 is a structural schematic diagram of a computer device provided by an embodiment of the present application. The computer device shown in the above Figure 2 or Figure 4 The computer device in the corresponding embodiment can be the computer device shown in the above Figure 7 As shown in the above Figure 7 The computer device can include a processor 701, an input device 702, an output device 703 and a memory 704. The processor 701, the input device 702, the output device 703 and the memory 704 are connected through a bus 705. The memory 704 is used for storing a computer program, and the computer program includes program instructions. The processor 701 is used for executing the program instructions stored in the memory 704.

[0115] In an embodiment of the present application, the processor 701 executes the following operations by running the executable program code in the memory 704:

[0116] obtain feature information of an element included in a first image and feature information of an element included in a second image, the first image including an environmental image of a location collected by a data collection device, the second image being an image similar to the first image in an image library included in map data;

[0117] match the element in the first image and the element in the second image according to the feature information of the element included in the first image and the feature information of the element included in the second image, to obtain an element matching result;

[0118] update the image library based on the element matching result and the first image.

[0119] In an embodiment, the processor 701 is further configured to:

[0120] invoke a search network to obtain one or more images similar to the first image from candidate images included in the image library, each candidate image in the image library corresponding to one or more three-dimensional coordinates, and each three-dimensional coordinate corresponding to one or more two-dimensional coordinates;

[0121] determine the second image from the one or more images according to the three-dimensional coordinates corresponding to each image and the two-dimensional coordinates corresponding to each three-dimensional coordinate.

[0122] In an embodiment, when the processor 701 determines the second image from the one or more images according to the three-dimensional coordinates corresponding to each image and the two-dimensional coordinates corresponding to each three-dimensional coordinate, the processor 701 can be specifically configured to:

[0123] from the one or more three-dimensional coordinates corresponding to each image, obtain a target three-dimensional coordinate whose number of corresponding two-dimensional coordinates exceeds a number threshold;

[0124] obtain a first total number of two-dimensional coordinates corresponding to each of the images and a second total number of two-dimensional coordinates corresponding to the target three-dimensional coordinates;

[0125] determine a reconstruction rate of each of the images according to the first total number and the second total number;

[0126] take an image corresponding to which the reconstruction rate is greater than or equal to a target threshold value in the one or more images as a second image.

[0127] In an embodiment, the processor 701, when updating the map based on the element matching result and the first image, can be specifically configured to:

[0128] if the element matching result indicates that there is a target element in the elements included in the first image, update the image library using the first image, the target element being any element in the first image, and the second image not including the target element.

[0129] In an embodiment, the processor 701, when matching elements in the first image and elements in the second image according to feature information of elements included in the first image and feature information of elements included in the second image, obtains an element matching result, can be specifically configured to:

[0130] invoke an element matching network to perform feature fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image, to obtain first fusion features of the elements included in the first image and second fusion features of the elements included in the second image;

[0131] match the elements in the first image and the elements in the second image according to the first fusion features and the second fusion features, to obtain an element matching result.

[0132] In an embodiment, the processor 701, when invoking an element matching network to perform feature fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image, to obtain first fusion features of the elements included in the first image and second fusion features of the elements included in the second image, can be specifically configured to:

[0133] invoke a feature fusion layer of the element matching network to perform intra-image fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image, respectively, to obtain internal fusion features of the elements included in the first image and internal fusion features of the elements included in the second image;

[0134] fusing, by calling a feature fusion layer of the element matching network, feature information of an element included in the first image and feature information of an element included in the second image, to obtain cross-fusion features of the element included in the first image and cross-fusion features of the element included in the second image;

[0135] determining first fusion features of the element included in the first image according to the internal fusion features and the cross-fusion features of the element included in the first image;

[0136] determining second fusion features of the element included in the second image according to the internal fusion features and the cross-fusion features of the element included in the second image.

[0137] In an embodiment, the processor 701 is further configured to:

[0138] obtain a training sample set, the training sample set including positive sample pairs, negative sample pairs, and label information of each sample pair, the label information including feature information of an element in each sample of each sample pair, the positive sample pair including samples in which elements corresponding to a same entity exist, and the negative sample pair including samples in which elements corresponding to a same entity do not exist;

[0139] process the positive sample pairs and the negative sample pairs by calling an initial neural network to obtain predicted matching elements corresponding to each sample pair;

[0140] adjust network parameters of the initial neural network by using the label information of each sample pair and the predicted matching elements corresponding to each sample pair to obtain an element matching network.

[0141] It should be understood that, in the embodiments of the present application, the processor 701 can be a central processing unit (CPU), and the processor 701 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0142] The memory 704 can include read-only memory and random access memory, and provide instructions and data to the processor 701. A portion of the memory 704 can also include non-volatile random access memory.

[0143] The input device 702 can include a camera assembly and the like, and input a first image to the processor 701; the output device 703 can include a display and the like.

[0144] In a specific implementation, the processor 701, the input device 702, the output device 703 and the memory 704 described in the embodiments of the present application can perform the implementation manners described in all the above embodiments, and can also perform the implementation manners described in the above apparatuses, which will not be described here.

[0145] The embodiments of the present application provide a computer readable storage medium, which stores a computer program, the computer program includes program instructions, and the program instructions are executed by a processor to perform the steps performed in all the above embodiments.

[0146] The embodiments of the present application also provide a computer program product or a computer program, which includes computer instructions, the computer instructions are stored in a computer readable storage medium, and the computer instructions are executed by a processor of a computer device to perform the method in all the above embodiments.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM) and the like.

[0148] The above only discloses a preferred embodiment of the present application, and of course cannot limit the scope of the present application. Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be implemented, and equivalent changes made according to the claims of the present application still belong to the scope of the present application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining feature information of elements included in a first image and feature information of elements included in a second image, the first image comprising an environment image of a location collected by a data collection device, and the second image being an image similar to the first image in an image library included in map data; calling a feature fusion layer of an element matching network to perform intra-image fusion on the feature information of elements included in the first image and the feature information of elements included in the second image respectively, to obtain internal fusion features of elements included in the first image and internal fusion features of elements included in the second image; calling the feature fusion layer of the element matching network to perform inter-image fusion on the feature information of elements included in the first image and the feature information of elements included in the second image, to obtain cross fusion features of elements included in the first image and cross fusion features of elements included in the second image; determining first fusion features of elements included in the first image according to the internal fusion features and the cross fusion features of the elements included in the first image, and determining second fusion features of elements included in the second image according to the internal fusion features and the cross fusion features of the elements included in the second image; calling an optimal assignment layer of the element matching network to perform feature cross multiplication on the first fusion features and the second fusion features, to obtain a matching matrix between elements in the first image and elements in the second image; performing optimal transport calculation on the matching matrix by using an optimal transport algorithm to obtain an optimal assignment matrix, and determining an element matching result between elements in the first image and elements in the second image according to the optimal assignment matrix; updating the image library based on the element matching result and the first image.

2. The method of claim 1, wherein, The method further comprises: calling a retrieval network to obtain one or more images similar to the first image from candidate images included in the image library, each candidate image in the image library corresponding to one or more three-dimensional coordinates, and each three-dimensional coordinate corresponding to one or more two-dimensional coordinates; determining a second image from the one or more images according to the three-dimensional coordinates corresponding to each image and the two-dimensional coordinates corresponding to each three-dimensional coordinate.

3. The method of claim 2, wherein, The determination of the second image from the one or more images according to the three-dimensional coordinates corresponding to each image and the two-dimensional coordinates corresponding to each three-dimensional coordinate comprises: from the one or more three-dimensional coordinates corresponding to each image, obtaining target three-dimensional coordinates whose number of corresponding two-dimensional coordinates exceeds a number threshold; obtaining a first total number of two-dimensional coordinates corresponding to each image and a second total number of two-dimensional coordinates corresponding to the target three-dimensional coordinates; determining a reconstruction rate of each image according to the first total number and the second total number; taking an image corresponding to a reconstruction rate greater than or equal to a target threshold from the one or more images as the second image.

4. The method according to any one of claims 1 to 3, characterized in that, The updating of the map based on the element matching result and the first image comprises: If the element matching result indicates that the first image includes a target element, the image library is updated using the first image, the target element being any element in the first image and the second image not including the target element.

5. The method of claim 1, wherein, The method further includes: obtaining a training sample set, the training sample set including positive sample pairs, negative sample pairs, and label information of each sample pair, the label information including feature information of elements in each sample of each sample pair, the positive sample pairs including samples having elements corresponding to the same entity, and the negative sample pairs including samples not having elements corresponding to the same entity; calling an initial neural network to process the positive sample pairs and the negative sample pairs to obtain predicted matching elements corresponding to each sample pair; adjusting network parameters of the initial neural network using the label information of each sample pair and the corresponding predicted matching elements to obtain an element matching network.

6. An image processing apparatus characterized by comprising: includes: an obtaining unit, configured to obtain feature information of elements included in a first image and feature information of elements included in a second image, the first image including an environmental image of a location collected by a data collection device, and the second image being an image similar to the first image in an image library included in map data; a processing unit, configured to call a feature fusion layer of an element matching network to perform intra-image fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image respectively to obtain internal fusion features of the elements included in the first image and internal fusion features of the elements included in the second image; call the feature fusion layer of the element matching network to perform inter-image fusion on the feature information of the elements included in the first image and the feature information of the elements included in the second image to obtain cross fusion features of the elements included in the first image and cross fusion features of the elements included in the second image; determine first fusion features of the elements included in the first image according to the internal fusion features and the cross fusion features of the elements included in the first image; and determine second fusion features of the elements included in the second image according to the internal fusion features and the cross fusion features of the elements included in the second image; call an optimal allocation layer of the element matching network to perform feature cross multiplication on the first fusion features and the second fusion features to obtain a matching matrix between elements in the first image and elements in the second image; perform optimal transport calculation on the matching matrix using an optimal transport algorithm to obtain an optimal allocation matrix, and determine an element matching result between the elements in the first image and the elements in the second image according to the optimal allocation matrix; The processing unit is further configured to update the image library based on the element matching result and the first image.

7. A computer device, comprising: includes: a processor adapted to implement one or more computer programs; and and A computer storage medium storing one or more computer programs adapted to be loaded and executed by the processor to perform the image processing method according to any one of claims 1-5.

8. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, performs the image processing method according to any one of claims 1-5.

9. A computer program product, characterised in that, The computer program product comprises a computer program; the computer program is stored in a computer readable storage medium, and the computer program, when executed by a processor, performs the image processing method according to any one of claims 1-5.

Images