Image matching method, device, equipment, storage medium and computer program product

Abstract

This application discloses an image matching method, apparatus, device, storage medium, and computer program product, relating to the fields of artificial intelligence and data processing technology. The method includes: identifying a first image and a second image respectively to obtain n objects in the first image and m objects in the second image, where n and m are positive integers; matching the descriptive term sets corresponding to the n objects and the descriptive term sets corresponding to the m objects respectively to obtain a first matching result between the first image and the second image; wherein, the descriptive term set corresponding to each object contains descriptive terms used to describe the features of the object, and the first matching result is used to characterize whether there are identical objects in the first image and the second image. This application, by supporting image-level object matching, reduces the mismatch rate of objects compared to one-to-one object matching because it comprehensively considers the matching degree between various objects, thereby improving the object matching accuracy.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data processing technology, and in particular to an image matching method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Frequent map updates allow users to perceive changes in the real world in a timely manner, enabling them to make more accurate navigation plans, and automated road data updates are an important foundation for achieving frequent map updates.

[0003] In related technologies, newly acquired road data (such as image data) is compared with historical road data to obtain the data that needs to be updated. The historical road data is then updated based on this updated data. For example, for a first object in a newly acquired image and a second object in a historical image, the set of descriptive terms corresponding to the first object in the newly acquired image and the set of descriptive terms corresponding to the second object in the historical image are first obtained. These two sets of descriptive terms are then compared to determine if the second object matches the first object. If they match, the image data corresponding to the first object is automatically updated based on the image data corresponding to the second object. The set of descriptive terms includes at least one descriptive term used to describe the features of the object.

[0004] However, when there are multiple highly similar objects, one-to-one matching of objects results in a certain mismatch rate and low object matching accuracy. Summary of the Invention

[0005] This application provides an image matching method, apparatus, device, storage medium, and computer program product, which can reduce the mismatch rate of object matching and improve the matching accuracy. The technical solution is as follows:

[0006] According to one aspect of the embodiments of this application, an image matching method is provided, the method comprising:

[0007] The first image and the second image are identified respectively to obtain n objects in the first image and m objects in the second image, where n and m are positive integers;

[0008] Obtain the sets of descriptive terms corresponding to the n objects and the sets of descriptive terms corresponding to the m objects; wherein, the set of descriptive terms corresponding to each object contains at least one descriptive term used to describe the features of the object;

[0009] The sets of descriptive terms corresponding to the n objects and the sets of descriptive terms corresponding to the m objects are matched to obtain a first matching result between the first image and the second image; wherein, the first matching result is used to characterize whether there are the same objects in the first image and the second image.

[0010] According to one aspect of the embodiments of this application, an image matching apparatus is provided, the apparatus comprising:

[0011] The image recognition module is used to recognize the first image and the second image respectively, to obtain n objects in the first image and m objects in the second image, where n and m are positive integers;

[0012] The description item acquisition module is used to acquire the description item sets corresponding to the n objects and the description item sets corresponding to the m objects respectively; wherein, the description item set corresponding to each object contains at least one description item used to describe the features of the object;

[0013] The result acquisition module is used to match the descriptive item sets corresponding to the n objects and the descriptive item sets corresponding to the m objects respectively to obtain a first matching result between the first image and the second image; wherein, the first matching result is used to characterize whether there are the same objects in the first image and the second image.

[0014] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described image matching method.

[0015] The computer device is a terminal or a server.

[0016] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described image matching method.

[0017] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image matching method described above.

[0018] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0019] By matching the sets of descriptive terms corresponding to n objects in the first image and the sets of descriptive terms corresponding to m objects in the second image, it is possible to determine whether there are identical objects in the first and second images. This achieves image-level object matching. Since the matching degree between each object is comprehensively considered, it solves the mismatch caused by one-to-one object matching in related technologies when there are multiple highly similar objects, thereby reducing the mismatch rate and improving the object matching accuracy. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the implementation environment of a solution provided in one embodiment of this application;

[0022] Figure 2 This is a flowchart of an image matching method provided in one embodiment of this application;

[0023] Figure 3 This is a flowchart of a method for obtaining a set of descriptive items according to an embodiment of this application;

[0024] Figure 4 and Figure 5 A flowchart illustrating the method for obtaining descriptive items is shown as an example;

[0025] Figure 6 This is a schematic diagram of a matching descriptive item provided in one embodiment of this application;

[0026] Figure 7 This is a schematic diagram of a matching method for identification results provided in one embodiment of this application;

[0027] Figure 8 This is a flowchart of an image matching method provided in another embodiment of this application;

[0028] Figure 9 This is a schematic diagram of a method for aggregating a set of recognition results provided in one embodiment of this application;

[0029] Figure 10 This is a block diagram of an image matching apparatus provided in one embodiment of this application;

[0030] Figure 11 This is a block diagram of an image matching apparatus provided in another embodiment of this application;

[0031] Figure 12 This is a block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0033] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0034] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0035] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0036] The technical solution provided in this application relates to computer vision technology in artificial intelligence. It uses computer vision technology to identify images, obtain objects in the images, and acquire the recognition results corresponding to the objects, the set of descriptive terms corresponding to the images, and the set of descriptive terms corresponding to the objects. It then uses the acquired recognition results corresponding to the objects, the set of descriptive terms corresponding to the images, and the set of descriptive terms corresponding to the objects to achieve object matching.

[0037] The method provided in this application can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. This computer device can be a terminal such as a PC (Personal Computer), tablet computer, smartphone, desktop computer, wearable device, intelligent robot, or vehicle-mounted device; it can also be a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0038] The technical solutions provided in this application are applicable to any scenario requiring object matching, such as object matching scenarios, image matching scenarios, navigation systems, and automated road data updates. The technical solutions provided in this application can reduce the mismatch rate of objects and improve the accuracy of object matching.

[0039] In one example, such as Figure 1 As shown, taking a navigation system as an example, the system may include a terminal 10 and a server 20.

[0040] Terminal 10 can be an electronic device such as a mobile phone, desktop computer, tablet computer, PC, wearable device, or in-vehicle device. A client application for the target application can be installed on terminal 10. The target application can be an object matching application, an image matching application, a navigation application, a simulation learning application, a map creation application, etc., and this embodiment does not limit the scope of the application.

[0041] Server 20 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Server 20 is used to provide background services for the client of the target application in terminal 10. For example, server 20 can be the background server of the aforementioned target application (such as a navigation application).

[0042] Terminal 10 and server 20 can communicate via network 30.

[0043] In one example, taking an object recognition scenario, the user sends the first image and the second image to the server 20 through the client in the terminal 10 (such as the client of the target application mentioned above). Based on the first image and the second image, the server 20 obtains the matching relationship between n objects in the first image and m objects in the second image (i.e., the object matching result) and sends the matching relationship to the client. The client then displays the matching relationship to the user.

[0044] Of course, in some other embodiments, the image matching process described above can also be executed in the terminal 10, such as in the client of the target application described above. For example, after acquiring the first image and the second image, the client obtains the matching relationship (i.e., the object matching result) between n objects in the first image and m objects in the second image based on the first image and the second image, and displays the matching relationship to the user.

[0045] In another example, taking the scenario of automated road data updates as an example, the client in terminal 10 (such as the client of the target application mentioned above) sends the latest image data collected by terminal 10 to server 20. After obtaining the latest image data, server 20 retrieves historical image data and uses the image matching method provided in this application embodiment to match the latest image data and historical image data to obtain object matching results. Based on the object matching results, it updates the historical image data to obtain the latest version of navigation data. Server 20 sends the latest version of navigation data to the client, and the client displays the latest version of navigation data to the user.

[0046] Of course, in some other embodiments, the above-described image matching process can also be executed in the terminal 10, such as in the client of the target application described above. For example, after obtaining the latest image data and historical image data, the client uses the image matching method provided in this application embodiment to match the latest image data and historical image data to obtain object matching results. Then, based on the object matching results, the historical image data is updated to obtain the latest version of navigation data, and the latest version of navigation data is displayed to the user.

[0047] Please refer to Figure 2 It illustrates a flowchart of an image matching method provided in an embodiment of this application, wherein the execution entity of each step of the method may be... Figure 1 The terminal 10 or server 20 in the implementation environment of the scheme shown may include the following steps (201~203).

[0048] Step 201: Identify the first image and the second image respectively to obtain n objects in the first image and m objects in the second image, where n and m are positive integers.

[0049] The first image and the second image can be two different images or two identical images. For example, the first image and the second image can refer to two images with different angles, different clarity, different content, or different generation times. This application embodiment does not limit the relationship between the first image and the second image. For example, in an object recognition scenario, the first image and the second image can refer to two images of the same object taken from different angles. In a road data automated update scenario, the first image and the second image can refer to two images of the same object taken at different times.

[0050] The aforementioned objects can refer to things such as people, animals, buildings, traffic elements (such as roads, signs, etc.), plants, etc., and this application embodiment does not limit this. In some embodiments, objects can also be referred to as physical points.

[0051] Optionally, an image recognition model or object detection model can be used to identify the first and second images, recognizing n objects in the first image and m objects in the second image. n and m can be the same or different. The aforementioned image recognition model or object detection model can be constructed using Faster R-CNN (a deep learning-based object detection method), SSD (Single Shot MultiBox Detector, an object detection method), YOLO (You Only LookOnce, a deep neural network-based object recognition and localization algorithm), etc. For example, refer to... Figure 1 The object detection model 40 identifies n objects in the first image and m objects in the second image. This identification result can be used to characterize the objects. The identification result can include the region of the object in the image (identified by a circumscribed rectangle or polygon), the type of the object, and the content of the object (such as the speed limit value in a speed limit sign). Based on the identification results corresponding to the n objects, n objects are obtained.

[0052] Step 202: Obtain the descriptive term set corresponding to each of the n objects and the descriptive term set corresponding to each of the m objects; wherein, the descriptive term set corresponding to each object contains at least one descriptive term used to describe the features of the object.

[0053] A descriptive term set refers to a collection of descriptive terms. The descriptive term set corresponding to an object is used to represent the object and can be obtained by combining the descriptive terms corresponding to the object. The descriptive term set corresponding to an image is used to represent the image and can be obtained by combining the descriptive terms corresponding to the image. A descriptive term is a data structure (such as a feature vector) used to describe a feature. For example, a descriptive term can be used to describe the feature points (such as detectors, also known as corners) corresponding to an object in an image. Feature points can refer to prominent pixels in the image, such as contour points, bright spots, and dark spots. The descriptive term in this application can also be called a descriptor. Optionally, for a feature point, mathematical features can be used to describe its surrounding features (such as its relative position to other pixels), forming a neighborhood feature vector centered on that feature point, which is the descriptive term corresponding to that feature point. Descriptive terms can also be extracted from feature points based on neural networks. For example, see [reference]. Figure 1 The descriptive term extraction model 50 extracts descriptive terms from the first and second images respectively, obtaining the descriptive term sets corresponding to the first and second images. This descriptive term extraction model 50 can be built based on algorithms such as SuperPoint (a method for extracting feature points and descriptive terms by training a network in a self-supervised manner) or D2-NET (a method for extracting feature points and descriptive terms based on a convolutional neural network). The descriptive terms extracted by the neural network can be used to represent the local semantic features of the image, describing a wider range of features and a broader context. Compared to descriptive terms obtained through rules or manual design, descriptive term extraction via neural networks is more efficient and accurate, thus improving the accuracy and efficiency of object matching.

[0054] In one example, taking the acquisition of the set of descriptive terms corresponding to an object through a neural network as an example, step 202 may also include the following sub-steps:

[0055] Step 202a: Perform feature extraction on the first image to obtain a first feature matrix, and perform feature extraction on the second image to obtain a second feature matrix.

[0056] In this embodiment of the application, the first feature matrix is ​​used to characterize the first image, and the second feature matrix is ​​used to characterize the second image. The dimensions of the first feature matrix and the second feature matrix can be adaptively set and adjusted according to actual usage.

[0057] Since the methods for obtaining the sets of descriptive terms corresponding to m objects and the sets of descriptive terms corresponding to n objects are the same, steps 202a to 202d will be explained using the method for obtaining the sets of descriptive terms corresponding to n objects as an example.

[0058] For example, refer to Figure 4 The process of obtaining the first feature matrix can be as follows: The first image 401 is input into the feature extraction network 402 (the feature extraction network corresponding to the description term extraction model). The feature extraction network 402 performs feature extraction on the first image 401 to obtain the first feature matrix 403. Here, the feature extraction network 402 can refer to a convolutional neural network. The feature extraction network 402 performs convolution on the first image 401 through convolutional layers to obtain W. H The three-dimensional feature map of C (that is, the dimension of the first feature matrix 403 is W) H C), W and H are the feature map dimensions, C is the number of channels in the feature map, and W, H and C are positive integers.

[0059] Step 202b: Based on the first feature matrix, obtain the feature point set corresponding to the first image, and based on the second feature matrix, obtain the feature point set corresponding to the second image; wherein, the feature points in the feature point set are used to describe the boundary information in the image.

[0060] In this embodiment, the feature point set refers to a collection of feature points. The feature point set may include multiple feature points. Boundary information may refer to contour information, edge information, etc.

[0061] In one example, the dimension of the first feature matrix is ​​W. H Taking C as an example, the process of obtaining the feature point set corresponding to the first image can be as follows:

[0062] 1. Divide the first feature matrix into multiple sub-feature matrices according to a set length in the W and H dimensions.

[0063] The set length can be determined based on empirical values, and this embodiment of the application does not impose any limitations on it. For example, refer to... Figure 5 Let's take a length of 3 as an example for explanation. In the W and H dimensions, with a set length of 3, the first feature matrix 501 is divided, resulting in multiple dimensions of 3. 3 The sub-feature matrix of C. Where W and H are not divisible by 3, the first feature matrix 501 is expanded along the dimensions of W and H, and all the expanded elements are set to 0 so that W and H are divisible by 3.

[0064] 2. For the target sub-feature matrix in multiple sub-feature matrices, divide the target sub-feature matrix in dimension C to obtain multiple first-class sub-feature matrices corresponding to the target sub-feature matrix.

[0065] Here, the target sub-feature matrix can refer to any one of multiple sub-feature matrices. For example, refer to... Figure 5 For the target sub-feature matrix 502, divide the target sub-feature matrix 502 into C units (e.g., 1) along the C dimension, resulting in C dimensions of 3. 3 The first type of sub-feature matrix of 1.

[0066] 3. Obtain the maximum points in each first-class sub-feature matrix and generate the first set of coordinate points.

[0067] Optionally, for the target first-class sub-feature matrix 503 among the C first-class sub-feature matrices, the maximum points in the target first-class sub-feature matrix 503 are obtained, and the maximum points in the C first-class sub-feature matrices are combined to generate a first coordinate point set.

[0068] 4. Divide the target sub-feature matrix in the W and H dimensions to obtain multiple second-class sub-feature matrices corresponding to the target sub-feature matrix.

[0069] Optionally, for the target sub-feature matrix 502, the target sub-feature matrix 502 is divided into units of length (e.g., 1) along both the W and H dimensions, resulting in 9 dimensions of 1. 1 The second type of sub-feature matrix of C.

[0070] 5. Obtain the maximum points in each second-class sub-feature matrix and generate the second set of coordinate points.

[0071] Optionally, for the target second-class sub-feature matrix 504 among the nine second-class sub-feature matrices, the maximum points in the target second-class sub-feature matrix 504 are obtained, and the maximum points in the nine second-class sub-feature matrices are combined to generate a second set of coordinate points.

[0072] 6. Based on the first set of coordinate points and the second set of coordinate points, obtain the set of feature points corresponding to the target sub-feature matrix.

[0073] If a coordinate point in the target sub-feature matrix has a value that is the maximum in both the horizontal and vertical dimensions, then that coordinate point can be considered a feature point, and its coordinates are the coordinates of that feature point. Therefore, the set of feature points corresponding to the target sub-feature matrix can be obtained by finding the intersection of the first set of coordinate points and the second set of coordinate points.

[0074] 7. Combine the feature point sets corresponding to the multiple sub-feature matrices to obtain the feature point set corresponding to the first image.

[0075] Alternatively, ((extended W) / 3) can be used. The feature point sets corresponding to the ((expanded H) / 3) sub-feature matrices are combined to obtain the feature point set corresponding to the first image.

[0076] Using the same method as for the first image, the set of feature points corresponding to the second image can be obtained.

[0077] It should be noted that, Figure 4 and Figure 5 This is merely illustrative and explanatory, and does not constitute a limitation on the content of this application.

[0078] Step 202c: Based on the feature point set corresponding to the first image, obtain the descriptive term set corresponding to the first image from the first feature matrix; and based on the feature point set corresponding to the second image, obtain the descriptive term set corresponding to the second image from the second feature matrix.

[0079] Optionally, the process of obtaining the descriptive term set corresponding to the first image can be as follows: for the target feature point in the feature point set corresponding to the first image, generate the descriptive term corresponding to the target feature point based on the second type of sub-feature matrix where the target feature point is located; combine the descriptive terms corresponding to each feature point in the feature point set corresponding to the first image to obtain the descriptive term set corresponding to the first image.

[0080] Here, the target feature point can refer to any feature point in the feature point set. The second type of sub-feature matrix containing the target feature point (e.g., with a dimension of 1) can be directly used. 1 The second type of sub-feature matrix of C is used as the descriptor for the target feature point.

[0081] Using the same method as for the first image, the set of descriptive terms corresponding to the second image can be obtained.

[0082] Step 202d: Divide the set of descriptive terms corresponding to the first image based on the n objects to obtain the set of descriptive terms corresponding to each of the n objects; and divide the set of descriptive terms corresponding to the second image based on the m objects to obtain the set of descriptive terms corresponding to each of the m objects.

[0083] Optionally, for the first object among the n objects, the region corresponding to the recognition result of the first object is expanded to obtain an expanded region; the descriptive terms in the expanded region are combined to obtain the descriptive term set corresponding to the first object. The same method can be used to obtain the descriptive term sets corresponding to each of the n objects. Theoretically, the total number of descriptive terms in the descriptive term sets corresponding to the n objects is greater than the number of descriptive terms in the descriptive term set corresponding to the first image.

[0084] Here, the first object can refer to any one of the n objects. The region corresponding to the recognition result refers to the area of ​​the object in the image. For example, the region corresponding to the recognition result of the first object refers to the area of ​​the first object in the first image. Optionally, the length and width of this region can be expanded by a set amount. For example, based on the original region, the length and width of the original region can be expanded by 20% respectively. This application does not limit the expansion method of the region corresponding to the recognition result.

[0085] Using the same method as for n objects, we can obtain the sets of descriptive terms corresponding to each of the m objects.

[0086] Step 203: Match the sets of description terms corresponding to the n objects and the sets of description terms corresponding to the m objects to obtain the first matching result between the first image and the second image; wherein, the first matching result is used to characterize whether there are the same objects in the first image and the second image.

[0087] In this embodiment, the first matching result includes the matching relationship between any one of the n objects and any one of the m objects. This matching relationship is used to characterize whether the two objects are the same object. For example, for a first object among the n objects and a second object among the m objects, if the first matching result indicates that the first object and the second object match, then the first object and the second object match based on the set of descriptive terms, and it can be determined that the first object and the second object may be the same object. If, at the same time, the matching rate between the first object and the second object is greater than the matching rate between the first object and other objects among the m objects, then it can be determined that the first object and the second object are the same object. Otherwise, it is determined that the first object and the second object are not the same object.

[0088] Optionally, if the first matching result indicates that there is no match between the first object and the second object, then the first object and the second object are different objects.

[0089] In one example, the process of obtaining the first matching result can be as follows:

[0090] 1. Calculate the matching rate between the set of descriptors corresponding to any object among n objects and the set of descriptors corresponding to any object among m objects, and obtain the matching rate set; wherein the matching rate set includes n×m matching rates.

[0091] The matching rate is used to characterize the degree of matching between objects. The specific process is as follows: For a first object out of n objects and a second object out of m objects, obtain a first value between the first and second objects. This first value refers to the number of matching descriptive terms between the descriptive term set corresponding to the first object and the descriptive term set corresponding to the second object. Obtain the first number of descriptive terms and the second number of descriptive terms. The first number of descriptive terms refers to the number of descriptive terms in the descriptive term set corresponding to the first object, and the second number of descriptive terms refers to the number of descriptive terms in the descriptive term set corresponding to the second object. Based on the first value, the first number of descriptive terms, and the second number of descriptive terms, calculate the matching rate between the descriptive term set corresponding to the first object and the descriptive term set corresponding to the second object.

[0092] Matchable descriptors are used to characterize pairs of descriptors that can match each other. For example, for a first target descriptor in the set of descriptors corresponding to a first object, a second target descriptor in the set of descriptors corresponding to a second object is found to be closest to the first target descriptor (e.g., with the highest similarity, denoted as d1). The first and second target descriptors are then combined into a descriptor pair; that is, the first and second target descriptors are matchable descriptors between the set of descriptors corresponding to the first object and the set of descriptors corresponding to the second object. This process is repeated to determine the set of descriptor pairs between the set of descriptors corresponding to the first object and the set of descriptors corresponding to the second object, and then incorrect descriptor pairs are removed. For example, for the first and second target descriptions, the third target description is found to be the second closest to the first target description from the set of descriptions corresponding to the second object. The distance between the first and third target descriptions is denoted as d2. If d1 / d2 is less than a threshold (e.g., 0.8), the first and second target descriptions are determined to be a correct description pair; otherwise, they are determined to be an incorrect description pair and deleted. This process is repeated to obtain a filtered set of description pairs. The number of descriptions in this filtered set is the number of matching descriptions between the set of descriptions corresponding to the first object and the set of descriptions corresponding to the second object. For example, refer to... Figure 6 For the buildings in the first image 601 and the buildings in the second image 602, the description items at both ends of the line connecting the two buildings are the description item pairs between the two buildings, and the number of lines is the number of matching description items between the two buildings.

[0093] Optionally, the RANSAC (Random Sample Consensus) method can also be used to remove erroneous descriptor pairs. Alternatively, the method of d1 / d2 being less than a threshold and the RANSAC method can be combined simultaneously to remove erroneous descriptor pairs; this embodiment of the application does not limit this approach. Optionally, the same method can be used to obtain the number of matching descriptors between the descriptor set corresponding to the first image and the descriptor set corresponding to the second image.

[0094] After obtaining the first numerical value, the number of the first and second descriptive items are summed to obtain the first sum value; the difference between the first sum value and the first numerical value is obtained to obtain the first difference value; the first numerical value and the first difference value are divided to obtain the matching rate between the set of descriptive items corresponding to the first object and the set of descriptive items corresponding to the second object.

[0095] The matching rates between the set of descriptors corresponding to any object among n objects and the set of descriptors corresponding to any object among m objects are combined to obtain a set of matching rates.

[0096] For example, let the set of descriptive terms corresponding to each of the n objects be: The set of descriptive terms corresponding to each of the m objects is as follows: For each of the n objects, the set of descriptive terms... The sets of descriptive terms corresponding to the m objects respectively , and If the number of matching descriptors between them is k, then and The matching rate between them is:

[0097] ;

[0098] in, for The number of corresponding descriptive items for The number of corresponding descriptive items.

[0099] Obtain each ( )and The matching rate between them ( )and The matching rates between the objects are combined to obtain a set of matching rates between the n objects and the m objects.

[0100] 2. Based on the matching rate set, determine the matching relationship between n objects and m objects to obtain the first matching result between the first image and the second image.

[0101] Optionally, based on the matching rate set, a bipartite graph is constructed between n objects and m objects; wherein the bipartite graph uses objects as nodes and the matching rate is the weight value of the connecting edge between nodes; the target complete match corresponding to the bipartite graph is obtained, the target complete match is the complete match with the largest sum of matching rates; based on the matching relationship between n objects and m objects in the target complete match, the first matching result is obtained.

[0102] A bipartite graph, in graph theory, is a graph model that divides the nodes in a graph into two disjoint subsets. It can also be called a bipartite graph, a binary graph, etc. For example, a bipartite graph between n objects and m objects can be a structured graph where n objects form one subset and m objects form another. The n objects and m objects can be connected across subsets via edges.

[0103] The Kuhn-Munkras (KM) algorithm (a weighted bipartite graph matching algorithm) can be used to match the bipartite graphs between n objects and m objects to obtain a perfect match. In a perfect match, any object among the n objects can find a matching object among the m objects, and any object among the m objects can also find a matching object among the n objects. If, in a perfect match, all connecting edges satisfy... If , then the perfect match is the target perfect match. Let be the weight value of the i-th object out of n objects. Let be the weight value of the j-th object out of m objects. This represents the weight value of the connection edge between the i-th object and the j-th object.

[0104] Optionally, the matching relationships between n objects and m objects in the target complete match can be combined to obtain a set of matching relationships, which is the first matching result.

[0105] In one example, based on the plane assumption, the transition matrix can also be used to match the set of descriptors corresponding to the first object and the set of descriptors corresponding to the second object. For example, see reference... Figure 7For the first object 703 in the first image 701 and the second object 704 in the second image 702, based on the matching relationship between the descriptive terms of the recognition results corresponding to the first object 703 and the recognition results corresponding to the second object 704 (e.g., generating a transition matrix), the bounding rectangle corresponding to the first object 703 can be transformed and projected onto the second image 702. If the projected bounding rectangle overlaps with the bounding rectangle corresponding to the second object 704, or if the overlap is greater than a certain threshold, then it is considered that the bounding rectangle corresponding to the first object 703 matches the bounding rectangle corresponding to the second object 704, that is, the descriptive term set corresponding to the first object matches the descriptive term set corresponding to the second object, and thus the first object and the second object match.

[0106] In one example, a second matching result can also be obtained between the first image and the second image. This second matching result is used to characterize whether the image scene corresponding to the first image and the image scene corresponding to the second image are the same. For example, if the second matching result indicates that the image scene corresponding to the first image and the image scene corresponding to the second image are the same, then for the first object in the first image and the second object in the second image, it can be determined that the first object and the second object match in the image scene.

[0107] Optionally, the process of obtaining the second matching result can be as follows: Obtain a second numerical value between the first image and the second image, where the second numerical value refers to the number of matchable descriptive terms between the descriptive term set corresponding to the first image and the descriptive term set corresponding to the second image; obtain the maximum value between the third number of descriptive terms and the fourth number of descriptive terms; wherein the third number of descriptive terms refers to the number of descriptive terms in the descriptive term set corresponding to the first image, and the fourth number of descriptive terms refers to the number of descriptive terms in the descriptive term set corresponding to the second image. Divide the second numerical value by the maximum value to obtain the matching rate between the first image and the second image; based on the matching rate between the first image and the second image, obtain the second matching result between the first image and the second image.

[0108] For example, let the first image be L1, the second image be L2, and the number of matching descriptors between the set of descriptors corresponding to the first image and the set of descriptors corresponding to the second image be t. Then the matching rate between the first image and the second image is:

[0109] ;

[0110] in, The number of description items corresponding to L1, The number of descriptors corresponding to L2 is denoted by DI, which represents the set of descriptors corresponding to the acquired image.

[0111] like If the value is greater than the threshold, then the image scene corresponding to the first image is considered to be the same as the image scene corresponding to the second image. If the threshold is less than or equal to the threshold, the image scene corresponding to the first image is considered to be different from the image scene corresponding to the second image. This threshold can be adaptively set and adjusted according to actual usage requirements.

[0112] In one example, a third matching result can also be obtained between objects, which is used to characterize whether the recognition results corresponding to the objects are the same. For example, if the third matching result indicates that the recognition result corresponding to the first object is the same as the recognition result corresponding to the second object, then it can be determined that the first object and the second object match in terms of recognition results.

[0113] Optionally, the process of obtaining the third recognition result can be as follows: for the first object among n objects and the second object among m objects, obtain the recognition result corresponding to the first object and the recognition result corresponding to the second object; wherein, the recognition result may include: the region of the object in the image, the type of the object and the content of the object; match the recognition result corresponding to the first object and the recognition result corresponding to the second object to obtain a third matching result between the first object and the second object, which is used to characterize whether the recognition result corresponding to the first object and the recognition result corresponding to the second object are the same.

[0114] For example, if the type of the first object is the same as the type of the second object, and the content of the first object (such as text, image, etc.) is the same as the content of the second object, then it can be determined that the recognition result corresponding to the first object is the same as the recognition result corresponding to the second object; otherwise, it can be determined that the recognition result corresponding to the first object is different from the recognition result corresponding to the second object. Optionally, based on this, the regions of the first object in the first image and the regions of the second object in the second image can also be matched. If the type of the first object is the same as the type of the second object, the content of the first object (such as text, image, etc.) is the same as the content of the second object, and the regions of the first object in the first image and the regions of the second object in the second image match, then it can be determined that the recognition result corresponding to the first object is the same as the recognition result corresponding to the second object; otherwise, it can be determined that the recognition result corresponding to the first object is different from the recognition result corresponding to the second object.

[0115] Using the same method as for the first and second objects, a third matching result can be obtained between any object among n objects and any object among m objects.

[0116] Optionally, for a first object in a first image and a second object in a second image, after obtaining the first matching result, the second matching result, and the third matching result, it can be determined whether the first object and the second object are the same based on the first matching result. For example, if the first matching result indicates that there is a matching relationship between the first object and the second object, and the matching rate between the first object and the second object is greater than the matching rate between the first object and other objects in the second image, it can be determined that the first object and the second object are the same.

[0117] Alternatively, the first matching result can be combined with the second matching result, or the first matching result can be combined with the third matching result, or the first matching result can be combined with the second and third matching results to determine whether the first object and the second object are the same. For example, if the first matching result indicates that the first object and the second object are the same, and the second matching result also indicates that the first object and the second object are a match, then the first object and the second object are determined to be the same; otherwise, the first object and the second object are determined to be different.

[0118] If the first matching result indicates that the first object and the second object are the same, and the third matching result also indicates that the first object and the second object are the same, then the first object and the second object are determined to be the same; otherwise, the first object and the second object are determined to be different.

[0119] If the first matching result indicates that the first object and the second object are the same, and if the second matching result and the third matching result both indicate that the first object and the second object are matched, then the first object and the second object are determined to be the same; otherwise, the first object and the second object are determined to be different.

[0120] By combining the first, second, and third matching results, the accuracy of object matching can be further improved.

[0121] In summary, the technical solution provided in this application achieves image-level object matching by matching the sets of descriptive terms corresponding to n objects in the first image and the sets of descriptive terms corresponding to m objects in the second image to determine whether there are identical objects in the first and second images. Since the matching degree between each object is comprehensively considered, the mismatch caused by one-to-one object matching in related technologies when there are multiple highly similar objects is solved, thereby reducing the mismatch rate of objects and improving the accuracy of object matching.

[0122] Furthermore, based on the matching rate set between the first and second images, the KM algorithm is used to obtain the matching relationship between n objects in the first image and m objects in the second image from a globally optimal perspective, which helps to improve the matching accuracy of objects.

[0123] Furthermore, extracting descriptive terms based on neural networks yields more accurate terms compared to rule-based or manually designed methods, thereby reducing the mismatch rate and improving matching accuracy. Simultaneously, it increases the efficiency of descriptive term acquisition, further enhancing object matching efficiency.

[0124] Furthermore, by combining the first matching result with the second and / or third matching result to perform object matching, the accuracy of object matching can be further improved.

[0125] In one exemplary embodiment, the application of the image recognition method described in the above embodiments in the scenario of automated road data updating is described in detail.

[0126] Please refer to Figure 8 It illustrates a flowchart of an image matching method provided in another embodiment of this application, wherein the execution entity of each step of the method may be... Figure 1 The terminal 10 or server 20 in the implementation environment of the scheme shown may include the following steps (801~809).

[0127] Step 801: Obtain the acquired image data and the task image set corresponding to the acquired image data.

[0128] Optionally, the aforementioned acquired image data may refer to image data obtained in the latest road data acquisition task. For example, a data acquisition vehicle takes pictures of a certain area (such as a road) in a certain direction of travel to obtain the corresponding acquired image data. The task image set includes multiple task images, which may refer to images obtained by processing the acquired image data (such as cropping, segmenting, etc.).

[0129] Step 802: Identify each task image in the task image set to obtain the candidate object set corresponding to the task image set.

[0130] Optionally, the candidate object set refers to the set obtained by combining the objects identified in each task image. The candidate object set may contain identical objects. These objects can refer to traffic elements on the road, such as traffic lights, traffic signs (e.g., speed limit signs), ground guidance information, and traffic facilities (e.g., traffic control towers). They can also refer to objects near the road, such as buildings and green vegetation; this embodiment of the application does not limit this.

[0131] Optionally, by recognizing each task image in the task image set, the recognition results corresponding to each object in the candidate object set can also be obtained. These recognition results may include the object's region in the image, the object's type, and the object's content. The image recognition method is the same as described in the above embodiments. Details not described in the embodiments of this application can be referred to in the above embodiments, and will not be repeated here.

[0132] Step 803: Aggregate the objects in the candidate object set to obtain the target object set and the recognition result set corresponding to each target object in the target object set.

[0133] Each target object in the target object set is unique. For the first target object in the target object set, all recognition results in the corresponding recognition result set are used to represent the first target object. Each recognition result in the recognition result set corresponding to the first target object corresponds to a different task image. In other words, the recognition results of the first target object in the task image including the first target object are combined to obtain the recognition result set corresponding to the first target object.

[0134] In one example, the aggregation process of the recognition result set can be as follows: Based on the image matching method described in the above embodiments, the target task image including the first target object is matched with each task image in the candidate task image set to obtain the recognition result set corresponding to the first target object. The candidate task image set can refer to a set composed of other task images in the task image set, or it can refer to a set after filtering the task image set based on the geographical location of the target task image. For example, task images within the vicinity of the target task image can be combined to obtain the candidate task image set, which includes multiple candidate task images.

[0135] For example, a set of descriptive terms corresponding to each object in the target task image and a set of descriptive terms corresponding to each object in each candidate task image are obtained. The set of descriptive terms corresponding to an object is used to describe that object. The descriptive terms in the set of descriptive terms can be extracted based on a neural network.

[0136] For each candidate task image, the descriptive term set corresponding to each object in the target task image is matched with the descriptive term set corresponding to each object in the candidate task image to obtain the first matching result between the target task image and the candidate task image. The same method is used to obtain the first matching result between the target task image and each candidate task image. The first matching result is used to characterize whether there are identical objects between the images.

[0137] Obtain the set of descriptive terms corresponding to the target task image, and the set of descriptive terms corresponding to each candidate task image. The set of descriptive terms corresponding to the task image is used to describe the task object.

[0138] For each target candidate task image, a second matching result is calculated based on the matching rate between the descriptor set corresponding to the target task image and the descriptor set corresponding to the target candidate task image. The same method is used to obtain the second matching result between the target task image and each candidate task image. The second matching result is used to characterize whether the image scenes corresponding to the images are identical.

[0139] Based on the second matching result, the candidate task image set is initially filtered to obtain the initially filtered candidate task image set. The image scene corresponding to each candidate task image in the initially filtered candidate task image set is the same as the image scene corresponding to the target task image.

[0140] Based on the first matching result, a second screening is performed on the candidate task image set after the initial screening to obtain a second-screened candidate task image set. Each candidate task image in the second-screened candidate task image set contains the same objects as the target task image.

[0141] Based on the first matching result, objects that have a matching relationship with the first target object are obtained from the candidate task image set after the second screening, and a candidate object set corresponding to the first target object is generated.

[0142] The recognition result corresponding to the first target object is matched with the recognition results corresponding to each candidate object in the candidate object set to obtain a third matching result between the first target object and each candidate object in the candidate object set. The third matching result is used to characterize whether the recognition results corresponding to the objects are the same.

[0143] The candidate object set is filtered based on the third matching result, resulting in a filtered candidate object set. The recognition result for each candidate object in the filtered candidate object set is the same as the recognition result for the first target object. After the above three filtering processes, it can be determined that the objects in the filtered candidate object set and the first target object are the same object.

[0144] The recognition results of each candidate object in the filtered candidate object set are combined to obtain the recognition result set corresponding to the first target object.

[0145] The image matching method in this embodiment is the same as that described in the above embodiments. For details not described in this embodiment, please refer to the above embodiments, which will not be repeated here.

[0146] In another example, refer to Figure 9 The aggregation process of the recognition result set can also be as follows: For the first target object 901, according to the shooting direction (movement trend), the area of ​​the first target object 901 in the image increases from small to large. Then, the objects in multiple consecutive task images that are on the same straight line as the first target object 901 are combined to obtain the object set corresponding to the first target object 901. The recognition results corresponding to each object in the object set corresponding to the first target object 901 are combined to obtain the recognition result set corresponding to the first target object 901.

[0147] Step 804: Select the target recognition result and the first image corresponding to each target object from the recognition result set corresponding to each target object in the target object set. The first image refers to the task image corresponding to the target recognition result.

[0148] Optionally, for the first target object, the target recognition result with the largest region (such as the circumscribed rectangle) in the recognition result set corresponding to the first target object can be determined as the target recognition result corresponding to the first target object.

[0149] Step 805: For the first target object in the target object set, obtain the second image corresponding to the first target object based on the geographical location of the first image corresponding to the first target object.

[0150] In this context, for the first target object, the second image refers to a historical image of the first target object included in a historical road data acquisition task. This historical road data acquisition task can refer to multiple past road data acquisition tasks to ensure that, during the automated updating of road data, each target object in the target object set can find corresponding matching data (such as image data) when matching data is required. Historical images can refer to images obtained by processing historically acquired image data (such as cropping, segmentation, etc.).

[0151] For example, the process of acquiring the second image can be as follows: For the first target object, based on the geographical location of the first image corresponding to the first target object, a candidate recognition result set corresponding to the first target object is obtained; wherein, the candidate recognition result in the candidate recognition result set refers to the recognition result in the historical road data collection task where the distance between the geographical location and the geographical location of the first image is less than a threshold (such as 30 meters); from the candidate recognition result set, a reference recognition result corresponding to the first target object is obtained, and the historical image corresponding to the reference recognition result corresponding to the first target object is determined as the second image corresponding to the first target object.

[0152] Optionally, the candidate recognition results in the candidate recognition result set can be sorted in chronological order to obtain a candidate recognition result sequence. The target recognition result corresponding to the first target object is then matched with each candidate recognition result in the candidate recognition result sequence to obtain a third matching result set. The third matching result in the third matching result set is used to characterize whether the target recognition result corresponding to the first target object matches the candidate recognition results. The first candidate recognition result in the candidate recognition result sequence that matches the target recognition result corresponding to the first target object is determined as the reference recognition result corresponding to the first target object.

[0153] Step 806: If the first target image has a corresponding second image, then proceed to step 807; otherwise, proceed to step 812.

[0154] Step 807: Match the sets of descriptive terms corresponding to the n objects in the first image and the sets of descriptive terms corresponding to the m objects in the second image to obtain a first matching result between the first image and the second image. This first matching result is used to characterize whether there are identical objects in the first image and the second image, where n and m are positive integers.

[0155] Step 808: If the first matching result represents a set of descriptive terms corresponding to the target recognition result and a set of descriptive terms corresponding to the reference recognition result, then proceed to step 809; otherwise, proceed to step 812.

[0156] Step 809: Based on the matching rate between the first image and the second image, a second matching result is obtained between the first image and the second image. This second matching result is used to characterize whether the image scene corresponding to the first image and the image scene corresponding to the second image are the same.

[0157] Step 810: If the second matching result indicates that the image scene corresponding to the first image matches the image scene corresponding to the second image, then proceed to step 811; otherwise, proceed to step 812.

[0158] Step 811: Based on the first image and the second image, automatically update the image data corresponding to the first target object.

[0159] Optionally, the image data corresponding to the first target object can be automatically updated based on the difference result obtained by performing differential processing on the first image and the second image.

[0160] Step 812: Transfer the task image set corresponding to the first target object to manual operation, and update the image data of the first target object through manual operation.

[0161] Optionally, the same method as for the first target object is used to update the image data of each target object in the target object set, thereby updating the road data.

[0162] In summary, the technical solution provided in this application achieves image-level object matching by matching the sets of descriptive terms corresponding to n objects in the first image and the sets of descriptive terms corresponding to m objects in the second image to determine whether there are identical objects in the first and second images. Since the matching degree between each object is comprehensively considered, the mismatch caused by one-to-one object matching in related technologies when there are multiple highly similar objects is solved, thereby reducing the mismatch rate of objects and improving the accuracy of object matching.

[0163] Furthermore, in the technical solution provided in this application embodiment, a more accurate descriptive term can be obtained through a neural network, thereby improving the object matching accuracy. Simultaneously, by combining the first matching result, the second matching result, and the third matching result to match objects, the object matching accuracy is further improved, which in turn improves the update accuracy of image data. Moreover, the improved object matching accuracy helps to increase the overall object matching rate, thereby reducing the amount of image data flowing into manual operations, allowing more image data to be updated automatically, and thus improving the update efficiency of road data.

[0164] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0165] refer to Figure 10 This diagram illustrates a block diagram of an image matching apparatus according to an embodiment of this application. The apparatus has the functionality to implement the method example described above; this functionality can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be located within a computer device. Figure 10 As shown, the device 1000 includes: an image recognition module 1001, a description item acquisition module 1002, and a result acquisition module 1003.

[0166] The image recognition module 1001 is used to recognize the first image and the second image respectively, to obtain n objects in the first image and m objects in the second image, where n and m are positive integers.

[0167] The description item acquisition module 1002 is used to acquire the description item sets corresponding to the n objects and the description item sets corresponding to the m objects respectively; wherein, the description item set corresponding to each object contains at least one description item for describing the features of the object.

[0168] The result acquisition module 1003 is used to match the descriptive item sets corresponding to the n objects and the descriptive item sets corresponding to the m objects respectively to obtain a first matching result between the first image and the second image; wherein, the first matching result is used to characterize whether there are the same objects in the first image and the second image.

[0169] In one exemplary embodiment, such as Figure 11 As shown, the result acquisition module 1003 includes: a matching rate acquisition submodule 1003a and a result acquisition submodule 1003b.

[0170] The matching rate acquisition submodule 1003a is used to calculate the matching rate between the set of descriptive terms corresponding to any object among the n objects and the set of descriptive terms corresponding to any object among the m objects, and obtain a matching rate set; wherein, the matching rate set includes n×m matching rates.

[0171] The result acquisition submodule 1003b is used to determine the matching relationship between the n objects and the m objects based on the matching rate set, and to obtain the first matching result between the first image and the second image.

[0172] In one exemplary embodiment, the result acquisition submodule 1003b is used for:

[0173] Based on the matching rate set, a bipartite graph is constructed between the n objects and the m objects; wherein the bipartite graph uses the objects as nodes and the matching rate is the weight value of the connecting edges between the nodes;

[0174] Obtain the target perfect matching corresponding to the bipartite graph, wherein the target perfect matching is the perfect matching with the largest sum of matching rates;

[0175] The first matching result is obtained based on the matching relationship between the n objects and the m objects in the target complete matching.

[0176] In one exemplary embodiment, the matching rate acquisition submodule 1003a is used for:

[0177] For the first object among the n objects and the second object among the m objects, obtain a first value between the first object and the second object. The first value refers to the number of matching descriptive items between the descriptive item set corresponding to the first object and the descriptive item set corresponding to the second object.

[0178] Obtain the first number of descriptive items and the second number of descriptive items; wherein, the first number of descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the first object, and the second number of descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the second object;

[0179] Based on the first value, the number of the first descriptive items, and the number of the second descriptive items, the matching rate between the set of descriptive items corresponding to the first object and the set of descriptive items corresponding to the second object is calculated.

[0180] The matching rate set is obtained by combining the matching rates between the set of descriptive terms corresponding to any object among the n objects and the set of descriptive terms corresponding to any object among the m objects.

[0181] In one exemplary embodiment, the matching rate acquisition submodule 1003a is further configured to:

[0182] The sum of the number of the first description items and the number of the second description items is used to obtain a first sum value;

[0183] The first difference is obtained by subtracting the first sum from the first numerical value.

[0184] Divide the first value by the first difference to obtain the matching rate between the set of descriptive terms corresponding to the first object and the set of descriptive terms corresponding to the second object.

[0185] In one exemplary embodiment, the result acquisition module 1003 is further configured to:

[0186] Obtain a second value between the first image and the second image, wherein the second value refers to the number of matching descriptive terms between the descriptive term set corresponding to the first image and the descriptive term set corresponding to the second image;

[0187] Obtain the maximum value between the number of third descriptive items and the number of fourth descriptive items; wherein, the number of third descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the first image, and the number of fourth descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the second image;

[0188] Divide the second value by the maximum value to obtain the matching rate between the first image and the second image;

[0189] Based on the matching rate between the first image and the second image, a second matching result is obtained between the first image and the second image; wherein, the second matching result is used to characterize whether the image scene corresponding to the first image and the image scene corresponding to the second image are the same.

[0190] In one exemplary embodiment, the result acquisition module 1003 is further configured to:

[0191] For the first object among the n objects and the second object among the m objects, obtain the recognition result corresponding to the first object and the recognition result corresponding to the second object; wherein, the recognition result includes: the region of the object in the image, the type of the object, and the content of the object;

[0192] The recognition results corresponding to the first object and the recognition results corresponding to the second object are matched to obtain a third matching result between the first object and the second object; wherein, the third matching result is used to characterize whether the recognition results corresponding to the first object and the recognition results corresponding to the second object are the same.

[0193] In one exemplary embodiment, such as Figure 11 As shown, the description item acquisition module 1002 includes: a feature extraction submodule 1002a, a feature point acquisition submodule 1002b, and a description item acquisition submodule 1002c.

[0194] The feature extraction submodule 1002a is used to extract features from the first image to obtain a first feature matrix, and to extract features from the second image to obtain a second feature matrix.

[0195] The feature point acquisition submodule 1002b is used to acquire a set of feature points corresponding to the first image based on the first feature matrix, and to acquire a set of feature points corresponding to the second image based on the second feature matrix; wherein the feature points in the feature point set are used to describe the boundary information in the image.

[0196] The description item acquisition submodule 1002c is used to acquire a set of description items corresponding to the first image from the first feature matrix based on the set of feature points corresponding to the first image, and to acquire a set of description items corresponding to the second image from the second feature matrix based on the set of feature points corresponding to the second image.

[0197] The description item acquisition submodule 1002c is further configured to divide the set of description items corresponding to the first image according to the n objects to obtain the set of description items corresponding to the n objects respectively, and to divide the set of description items corresponding to the second image according to the m objects to obtain the set of description items corresponding to the m objects respectively.

[0198] In one exemplary embodiment, the dimension of the first feature matrix is ​​W. H C, W, H, and C are positive integers; the feature point acquisition submodule 1002b is used for:

[0199] The first feature matrix is ​​divided into multiple sub-feature matrices according to a set length along the W and H dimensions.

[0200] For the target sub-feature matrix among the multiple sub-feature matrices, the target sub-feature matrix is ​​divided along the C dimension to obtain multiple first-class sub-feature matrices corresponding to the target sub-feature matrix;

[0201] Obtain the maximum points in each first-class sub-feature matrix and generate the first set of coordinate points;

[0202] The target sub-feature matrix is ​​divided along the W and H dimensions to obtain multiple second-class sub-feature matrices corresponding to the target sub-feature matrix;

[0203] Obtain the maximum points in each second-class sub-feature matrix and generate a set of second coordinate points;

[0204] Based on the first set of coordinate points and the second set of coordinate points, the set of feature points corresponding to the target sub-feature matrix is ​​obtained;

[0205] The feature point sets corresponding to the multiple sub-feature matrices are combined to obtain the feature point set corresponding to the first image.

[0206] In one exemplary embodiment, the description item acquisition submodule 1002c is further configured to:

[0207] For a target feature point in the feature point set corresponding to the first image, a description item corresponding to the target feature point is generated based on the second type of sub-feature matrix where the target feature point is located;

[0208] The descriptive terms corresponding to each feature point in the feature point set corresponding to the first image are combined to obtain the descriptive term set corresponding to the first image.

[0209] In one exemplary embodiment, the description item acquisition submodule 1002c is further configured to:

[0210] For the first object among the n objects, the region corresponding to the recognition result of the first object is expanded to obtain the expanded region;

[0211] The descriptive items in the expanded region are combined to obtain the set of descriptive items corresponding to the first object.

[0212] In summary, the technical solution provided in this application achieves image-level object matching by matching the sets of descriptive terms corresponding to n objects in the first image and the sets of descriptive terms corresponding to m objects in the second image to determine whether there are identical objects in the first and second images. Since the matching degree between each object is comprehensively considered, the mismatch caused by one-to-one object matching in related technologies when there are multiple highly similar objects is solved, thereby reducing the mismatch rate of objects and improving the accuracy of object matching.

[0213] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0214] Please refer to Figure 12 This diagram illustrates a structural block diagram of a computer device according to an embodiment of this application. This computer device can be used to implement the image matching method provided in the above embodiments. Specifically, it may include the following:

[0215] The computer device 1200 includes a central processing unit (such as a CPU, GPU, or FPGA) 1201, a system memory 1204 including RAM (Random-Access Memory) 1202 and ROM (Read-Only Memory) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The computer device 1200 also includes a basic input / output system (I / O system) 1206 to facilitate information transfer between various devices within the server, and a mass storage device 1207 for storing the operating system 1213, application programs 1214, and other program modules 1215.

[0216] The basic input / output system 1206 includes a display 1208 for displaying information and an input device 1209 for user input, such as a mouse or keyboard. Both the display 1208 and the input device 1209 are connected to the central processing unit 1201 via an input / output controller 1210 connected to the system bus 1205. The basic input / output system 1206 may also include the input / output controller 1210 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1210 also provides output to a display screen, printer, or other types of output devices.

[0217] The mass storage device 1207 is connected to the central processing unit 1201 via a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer-readable media provide non-volatile storage for the computer device 1200. That is, the mass storage device 1207 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0218] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1204 and mass storage device 1207 described above can be collectively referred to as memory.

[0219] According to an embodiment of this application, the computer device 1200 can also be connected to a remote computer on a network, such as the Internet, for operation. That is, the computer device 1200 can be connected to the network 1212 via the network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).

[0220] The memory further includes at least one instruction, at least one program, code set, or instruction set, wherein at least one instruction, at least one program, code set, or instruction set is stored in the memory and configured to be executed by one or more processors to implement the above image matching method.

[0221] In one exemplary embodiment, a computer-readable storage medium is also provided, the storage medium storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being executed by a processor to implement the above-described image matching method.

[0222] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0223] In one exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the image matching method described above.

[0224] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0225] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image matching method, characterized in that, The method includes: The first image and the second image are identified respectively to obtain n objects in the first image and m objects in the second image, where n and m are positive integers; Obtain the sets of descriptive terms corresponding to the n objects and the sets of descriptive terms corresponding to the m objects; wherein, the set of descriptive terms corresponding to each object contains at least one descriptive term used to describe the features of the object; For the first object among the n objects and the second object among the m objects, obtain a first value between the first object and the second object. The first value refers to the number of matching descriptive items between the descriptive item set corresponding to the first object and the descriptive item set corresponding to the second object. Obtain the first number of descriptive items and the second number of descriptive items; wherein, the first number of descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the first object, and the second number of descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the second object; Based on the first value, the number of the first descriptive items, and the number of the second descriptive items, the matching rate between the set of descriptive items corresponding to the first object and the set of descriptive items corresponding to the second object is calculated. The matching rates between the set of descriptive terms corresponding to any one of the n objects and the set of descriptive terms corresponding to any one of the m objects are combined to obtain a set of matching rates; wherein the set of matching rates includes n×m matching rates; Based on the matching rate set, the matching relationship between the n objects and the m objects is determined, and a first matching result between the first image and the second image is obtained; wherein, the first matching result is used to characterize whether there are the same objects in the first image and the second image.

2. The method according to claim 1, characterized in that, The step of determining the matching relationship between the n objects and the m objects based on the matching rate set, and obtaining a first matching result between the first image and the second image, includes: Based on the matching rate set, a bipartite graph is constructed between the n objects and the m objects; wherein the bipartite graph uses the objects as nodes and the matching rate is the weight value of the connecting edges between the nodes; Obtain the target perfect matching corresponding to the bipartite graph, wherein the target perfect matching is the perfect matching with the largest sum of matching rates; The first matching result is obtained based on the matching relationship between the n objects and the m objects in the target complete matching.

3. The method according to claim 1, characterized in that, The step of calculating the matching rate between the set of description items corresponding to the first object and the set of description items corresponding to the second object based on the first numerical value, the number of the first description items, and the number of the second description items includes: The sum of the number of the first description items and the number of the second description items is used to obtain a first sum value; The first difference is obtained by subtracting the first sum from the first numerical value. Divide the first value by the first difference to obtain the matching rate between the set of descriptive terms corresponding to the first object and the set of descriptive terms corresponding to the second object.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain a second value between the first image and the second image, wherein the second value refers to the number of matching descriptive terms between the descriptive term set corresponding to the first image and the descriptive term set corresponding to the second image; Obtain the maximum value between the number of third descriptive items and the number of fourth descriptive items; wherein, the number of third descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the first image, and the number of fourth descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the second image; Divide the second value by the maximum value to obtain the matching rate between the first image and the second image; Based on the matching rate between the first image and the second image, a second matching result is obtained between the first image and the second image; wherein, the second matching result is used to characterize whether the image scene corresponding to the first image and the image scene corresponding to the second image are the same.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Obtain the recognition result corresponding to the first object and the recognition result corresponding to the second object; wherein, the recognition result includes: the region of the object in the image, the type of the object, and the content of the object; The recognition results corresponding to the first object and the recognition results corresponding to the second object are matched to obtain a third matching result between the first object and the second object; wherein, the third matching result is used to characterize whether the recognition results corresponding to the first object and the recognition results corresponding to the second object are the same.

6. The method according to any one of claims 1 to 5, characterized in that, The step of obtaining the set of descriptive terms corresponding to the n objects and the set of descriptive terms corresponding to the m objects includes: Feature extraction is performed on the first image to obtain a first feature matrix, and feature extraction is performed on the second image to obtain a second feature matrix; Based on the first feature matrix, a set of feature points corresponding to the first image is obtained, and based on the second feature matrix, a set of feature points corresponding to the second image is obtained; wherein, the feature points in the set of feature points are used to describe the boundary information in the image; Based on the feature point set corresponding to the first image, a descriptive term set corresponding to the first image is obtained from the first feature matrix; and based on the feature point set corresponding to the second image, a descriptive term set corresponding to the second image is obtained from the second feature matrix. The set of descriptive terms corresponding to the first image is divided according to the n objects to obtain the set of descriptive terms corresponding to each of the n objects; and the set of descriptive terms corresponding to the second image is divided according to the m objects to obtain the set of descriptive terms corresponding to each of the m objects.

7. The method according to claim 6, characterized in that, The dimension of the first feature matrix is ​​W. H C, W, H and C are positive integers; The step of obtaining the feature point set corresponding to the first image based on the first feature matrix includes: The first feature matrix is ​​divided into multiple sub-feature matrices according to a set length along the W and H dimensions. For the target sub-feature matrix among the multiple sub-feature matrices, the target sub-feature matrix is ​​divided along the C dimension to obtain multiple first-class sub-feature matrices corresponding to the target sub-feature matrix; Obtain the maximum points in each first-class sub-feature matrix and generate the first set of coordinate points; The target sub-feature matrix is ​​divided along the W and H dimensions to obtain multiple second-class sub-feature matrices corresponding to the target sub-feature matrix; Obtain the maximum points in each second-class sub-feature matrix and generate a set of second coordinate points; Based on the first set of coordinate points and the second set of coordinate points, the set of feature points corresponding to the target sub-feature matrix is ​​obtained; The feature point sets corresponding to the multiple sub-feature matrices are combined to obtain the feature point set corresponding to the first image.

8. The method according to claim 6, characterized in that, The step of obtaining the set of descriptive terms corresponding to the first image from the first feature matrix based on the set of feature points corresponding to the first image includes: For a target feature point in the feature point set corresponding to the first image, a description item corresponding to the target feature point is generated based on the second type of sub-feature matrix where the target feature point is located; The descriptive terms corresponding to each feature point in the feature point set corresponding to the first image are combined to obtain the descriptive term set corresponding to the first image.

9. The method according to claim 6, characterized in that, The step of dividing the set of descriptive terms corresponding to the first image based on the n objects to obtain the set of descriptive terms corresponding to each of the n objects includes: For the first object among the n objects, the region corresponding to the recognition result of the first object is expanded to obtain the expanded region; The descriptive items in the expanded region are combined to obtain the set of descriptive items corresponding to the first object.

10. An image matching device, characterized in that, The device includes: The image recognition module is used to recognize the first image and the second image respectively, to obtain n objects in the first image and m objects in the second image, where n and m are positive integers; The description item acquisition module is used to acquire the description item sets corresponding to the n objects and the description item sets corresponding to the m objects respectively; wherein, the description item set corresponding to each object contains at least one description item used to describe the features of the object; The result acquisition module is used to: acquire a first value between the first object and the second object among the n objects and the m objects, where the first value refers to the number of matching descriptive items between the descriptive item set corresponding to the first object and the descriptive item set corresponding to the second object; acquire a first number of descriptive items and a second number of descriptive items, where the first number of descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the first object and the second number of descriptive items refers to the number of descriptive items in the descriptive item set corresponding to the second object; calculate the matching rate between the descriptive item set corresponding to the first object and the descriptive item set corresponding to the second object based on the first value, the first number of descriptive items, and the second number of descriptive items; combine the matching rate between the descriptive item set corresponding to any object among the n objects and the descriptive item set corresponding to any object among the m objects to obtain a matching rate set, where the matching rate set includes n×m matching rates; determine the matching relationship between the n objects and the m objects based on the matching rate set to obtain a first matching result between the first image and the second image; where the first matching result is used to characterize whether there are identical objects in the first image and the second image.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the image matching method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the image matching method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer instructions that are executed by a processor to implement the image matching method as described in any one of claims 1 to 9.

Images