Data processing method, device and system

Abstract

The present invention discloses a data processing method, device and system. The method comprises: obtaining a target image, wherein the target image includes a main body part and a non-main body part; obtaining second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of the main body in the target image; searching for a candidate image corresponding to the target image from an image library; obtaining first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of the main body in the candidate image; searching for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information. The present invention solves the technical problem of low accuracy in image matching in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a data processing method, device and system. Background Art

[0002] The problem of vehicle image search based on massive traffic surveillance video image data is receiving more and more attention. At present, deep neural networks are usually used to search for vehicles based on global features to determine the identity of a specified vehicle, such as the color, appearance, and brand of the vehicle. However, due to the fact that some vehicles in city-level surveillance data are very similar in appearance, the global feature image search technology based on deep neural networks is difficult to perform accurate search in some cases, making it difficult to effectively identify the identity of the vehicle.

[0003] With respect to the problem of low accuracy in image matching in the prior art, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, device and system to at least solve the technical problem of low accuracy in image matching in the prior art.

[0005] According to one aspect of an embodiment of the present invention, there is provided a data processing method, comprising: acquiring a target image, wherein the target image includes a main body part and a non-main body part; acquiring second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of the main body in the target image; searching for a candidate image corresponding to the target image from an image library; acquiring first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of the main body in the candidate image; and searching for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information.

[0006] According to another aspect of an embodiment of the present invention, a data processing method is also provided, including: obtaining a target image of a target vehicle, wherein the target vehicle is a vehicle that causes a traffic violation, and the target image is an image of the target vehicle taken when the traffic violation occurs; searching for a candidate vehicle corresponding to the target vehicle from a vehicle image library according to the target image; obtaining first local feature point information of the candidate vehicle in the candidate vehicle image; obtaining second local feature point information of the target image; and searching for a matching vehicle that matches the target vehicle from the candidate vehicles based on the first local feature point information and the second local feature point information.

[0007] According to another aspect of an embodiment of the present invention, a data processing device is also provided, including: a first acquisition module, used to acquire a target image, wherein the target image includes a main body part and a non-main body part; a second acquisition module, used to acquire second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of the main body in the target image; a first search module, used to search for a candidate image corresponding to the target image from an image library; a third acquisition module, used to acquire the first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of the main body in the candidate image; and a second search module, used to search for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information.

[0008] According to another aspect of an embodiment of the present invention, a data processing system is also provided, including: an image acquisition device for acquiring a target image; a processor for acquiring the target image and acquiring second local feature point information of the target image, searching for a candidate image corresponding to the target image from an image library, acquiring first local feature point information of the candidate image, and searching for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information, wherein the target image includes a main body part and a non-main body part, the first local feature point information includes a local feature point vector of the main body in the candidate image, and the second local feature point information includes a local feature point vector of the main body in the target image.

[0009] According to another aspect of an embodiment of the present invention, a storage medium is further provided. The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above-mentioned data processing method.

[0010] According to another aspect of an embodiment of the present invention, a processor is further provided, the processor is used to run a program, wherein the program executes the above-mentioned data processing method when it is run.

[0011] In an embodiment of the present invention, a target image is obtained, and second local feature point information of the target image is obtained, wherein the second local feature point information includes a local feature point vector of a subject in the target image, and a candidate image corresponding to the target image is searched from an image library; first local feature point information of the candidate image is obtained, wherein the first local feature point information is a local feature point vector of a subject in the candidate image; and based on the first local feature point information and the second local feature point information, a matching image that matches the target image is searched from the candidate image. The above scheme performs matching based on the local feature information of the candidate image and the local feature information of the target image, based on the preliminary extraction of the candidate image of the target image from the image library, so that the detailed information of the image can be perceived, and the accuracy of image matching can be improved, which solves the technical problem of low accuracy in image matching in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0013] Figure 1 A hardware structure block diagram of a computing device (or mobile device) for implementing a data processing method is shown;

[0014] Figure 2 is a flow chart of a data processing method according to Embodiment 1 of the present application;

[0015] Figure 3 is a schematic diagram of performing image search based on a vehicle image according to an embodiment of the present application;

[0016] Figure 4a is a schematic diagram of adjusting a first local feature point according to an embodiment of the present application;

[0017] Figure 4b is a schematic diagram of another method for adjusting the first local feature point according to an embodiment of the present application;

[0018] Figure 5 is a flow chart of a data processing method according to Embodiment 2 of the present application;

[0019] Figure 6 is a schematic diagram of an image matching system according to Embodiment 3 of the present application;

[0020] Figure 7 is a schematic diagram of a data processing device according to Embodiment 4 of the present application;

[0021] Figure 8is a schematic diagram of a data processing device according to Embodiment 5 of the present application;

[0022] Fig. 9 is a structural block diagram of a computing device according to Embodiment 6 of the present application

[0023] Fig.10 is a flow chart of a data processing method according to Embodiment 8 of the present application; and

[0024] Fig.11 It is a schematic diagram of a data processing device according to Example 9 of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Example 1

[0028] According to an embodiment of the present invention, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computing device or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device (or mobile device) for implementing a data processing method. Figure 1As shown, the computing device 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0030] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computing device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the vulnerability detection method of the above-mentioned application program. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computing device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of computing device 10 (or mobile device).

[0034] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0035] Under the above operating environment, this application provides Figure 2 The data processing method shown. Figure 2 is a flow chart of a data processing method according to Embodiment 1 of the present application. Figure 1 As shown, the method comprises the following steps:

[0036] Step S21, acquiring a target image, wherein the target image includes a main body part and a non-main body part.

[0037] Specifically, the target image may be an image containing a subject to be searched, the subject portion in the target image is used to indicate the portion of the subject to be searched in the target image, and the non-subject portion is used to indicate other portions except the subject portion to be searched. The subject to be searched may include: moving subjects such as vehicles, people, animals, and drones.

[0038] For example, the target image is an image taken at a traffic checkpoint and containing a target vehicle. The portion of the image where the target vehicle is located is the main portion, and the other portions except the target vehicle are the non-main portion.

[0039] For another example, taking searching for children or the elderly as an example, the target image may also be an image taken by a camera containing children or the elderly. The part of the image where the children or the elderly are located is the main part, and the other parts except the children or the elderly are the non-main part.

[0040] Step S23, obtaining second local feature point information of the target image, wherein the second local feature point information is a local feature point vector of the subject in the target image.

[0041] Specifically, the main body of the target image has multiple local feature points, and the second local feature point information is a set of vectors of multiple local feature points on the main body of the target image. Local feature points are used to describe local features, which are features extracted from local areas of the image, including edges, corners, lines, curves, and areas with special attributes. Compared with global image features such as line features, texture features, and structural features, local features of an image have the characteristics of being abundant in number, having low correlation between features, and not affecting the detection and matching of other features due to the disappearance of some features in the case of occlusion. Local features also have the characteristics of high feature detection repetition rate and fast speed, robust feature description to image transformations such as illumination, rotation, and viewpoint changes, low dimension of feature descriptors, and easy to achieve fast matching.

[0042] In an optional embodiment, taking the target image as an image of a traffic violation vehicle as an example, the local features of the target image can be extracted by a preset local feature point extraction model to obtain second local feature point information of the target image.

[0043] Step S25, searching the image library for a candidate image corresponding to the target image.

[0044] Specifically, the image library may be a database of a large number of images containing subjects, and in the image library, the subjects in the acquired images may be extracted and then stored. The candidate image may be an image obtained by preliminary screening from the image library based on one or more global features in the target image. The above global features may include any one or more of the following: color, model, brand, etc.

[0045] In an optional embodiment, images of vehicles captured by cameras at traffic checkpoints throughout the city are acquired in real time, and the vehicles in the images are extracted and stored in an image library. When a traffic checkpoint detects that there is an illegal vehicle, but the vehicle license plate is difficult to accurately identify due to angle, light or human factors, the image captured when the illegal vehicle violates the traffic rules is used as the target image, and the corresponding candidate image is searched in the image library based on the global features of the illegal vehicle.

[0046] Through the above steps, vehicle images similar to the vehicle in the target image can be screened out from the massive vehicle images stored in the image library, thereby excluding most vehicles. After the candidate images are obtained, further searches are performed in the candidate images.

[0047] Step S27: obtaining first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of the subject in the candidate image.

[0048] In an optional embodiment, the local features of the candidate image may still be extracted by using a preset local feature point extraction model to obtain the first local feature point information of the candidate image.

[0049] Step S29: searching for a matching image that matches the target image from the candidate images based on the first local feature point information and the second local feature point information.

[0050] In the above scheme, after obtaining the first local feature point information and the second local feature point information, the similarity between the two can be obtained, and the similarity is used to represent the similarity between the candidate image and the target image, so as to determine the matching image that best matches the target image according to the similarity.

[0051] In an optional embodiment, still taking the image of the illegal vehicle as the target image as an example, when the vehicle violates the traffic rules, it is difficult to accurately identify the vehicle license plate due to angle, light or human factors, and thus the identity of the vehicle cannot be identified. However, the vehicle will also leave images at other traffic checkpoints, and the images left at other traffic checkpoints may contain the vehicle license plate. In this case, by matching the local feature point information, the matching image that best matches the target image is found from the candidate images, and it is considered that the subject in the matching image is the above-mentioned illegal vehicle, and the identity of the vehicle can be determined based on the license plate in the matching image.

[0052] This solution uses image search to authenticate the identity of vehicles, which can significantly improve the search accuracy of the vehicle image search system when there are many vehicles of the same brand, providing a powerful means for the control and inspection of illegal vehicles in traffic scenes.

[0053] The above embodiment of the present application searches for a candidate image corresponding to a target image from an image library; obtains first local feature point information of the candidate image, wherein the first local feature point information is a local feature point vector of a subject in the candidate image; obtains second local feature point information of the target image, wherein the second local feature point information is a local feature point vector of a subject in the target image; and searches for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information. The above scheme performs matching based on the local feature information of the candidate image and the local feature information of the target image, based on the preliminary extraction of the candidate image of the target image from the image library, so that the detailed information of the image can be perceived, thereby improving the accuracy of image matching, and solving the technical problem of low accuracy in image matching in the prior art.

[0054] As an optional embodiment, searching for a candidate image corresponding to the target image from an image library includes: extracting global feature information of the target image; and searching for the candidate image in the image library based on the global feature information.

[0055] Specifically, the global feature information of the target image may include: color feature information, texture feature information, shape feature information, and the like.

[0056] Still taking the vehicle as an example, the global feature information can be the color, brand, shape, intensity histogram, etc. of the vehicle in the target image. By matching the global feature information, candidate images with a certain degree of similarity to the vehicle in the target image can be recalled from the image library, thereby excluding a large number of irrelevant vehicle images in the image library. More specifically, if the illegal vehicle in the target image is a red sedan, then all images of vehicles other than the red sedan in the image library will be excluded, leaving only the image of the red sedan as a candidate image.

[0057] As an optional embodiment, obtaining the first local feature point information of the candidate image includes: obtaining the first local feature point information of the candidate image by using a preset deep local feature point extraction model.

[0058] Specifically, the above-mentioned preset deep local feature extraction model is used to extract feature point information in the image. In an optional embodiment, the deep local feature point extraction model can be an encoder-decoder structure, and the feature point information can include: feature point position and feature point vector, so the above-mentioned deep local feature point extraction model has one input and two outputs, and the candidate image is input into the deep local feature point extraction model to obtain the feature point position and feature point vector output by the deep local feature point extraction model.

[0059] As an optional embodiment, obtaining second local feature point information of the target image includes: receiving a selection instruction, wherein the selection instruction is used to select a to-be-matched area in the target image; obtaining local feature point information of the to-be-matched area through a preset deep local feature point extraction model, and determining that the local feature point information of the to-be-matched area is the second local feature point information.

[0060] Specifically, the above-mentioned preset deep local feature point extraction model may be the same as the model used to extract the first local feature point information.

[0061] In an optional embodiment, the above selection instruction can be issued by the user to select the area to be matched in the target image. For example, if there is an area with obvious features in the main body of the target image, the user can select this area, so as to use the deep local feature point extraction model to extract local feature point information in the selected area, thereby making the extracted local feature point information more targeted. Still taking the target image as an image of an illegal vehicle as an example, if the target image captures a banner hung on the front of the illegal vehicle, the area where the banner is located can be selected as the area to be matched.

[0062] In another optional embodiment, the above-mentioned area to be matched can also be an area of ​​interest in the target image, and the selection instruction is issued by the processing model for extracting the area of ​​interest in the target image. Still taking the target image as an image of an illegal vehicle as an example, the processing model of the area of ​​interest extracts the area of ​​interest from the target image as the area to be matched.

[0063] Figure 3 is a schematic diagram of performing image search based on vehicle images according to an embodiment of the present application, combined with Figure 3 As shown in the figure, in the target image, there is a banner "XXXX" hanging above the vehicle, which is the difference between this vehicle and similar or even the same model of vehicles. On this basis, the user can select the area where the banner is located in the image to obtain the area to be matched. The deep local feature point extraction model will extract local feature point information from the selected area to be matched.

[0064] As an optional embodiment, the above method also includes: obtaining a deep local feature point extraction model, obtaining the deep local feature point extraction model includes: training based on key points in a preset two-dimensional graphic to obtain a basic key point detection model; detecting sample feature points on the subject in the sample image through the basic key point detection model; using the sample feature points as sample data, training a preset initial network model to obtain a deep local feature point extraction model, wherein the preset initial network model includes an encoder and two decoders, and the two decoders are respectively used to predict the position and vector of the feature points.

[0065] Specifically, the above-mentioned preset two-dimensional graphics can be a simple basic image, such as a triangle, a rectangle, etc. In an optional embodiment, taking a triangle as an example, the points where the three corners of the triangle are located can be used as feature points, and the two-dimensional graphics of the triangle and the positions and vectors of the points where the three corners are located can be used as sample data for training the key point detection model. The basic key point detection model is simulated and pre-trained to obtain a basic key point detection model.

[0066] The above sample images are different from images containing two-dimensional graphics, and they contain real subjects. Taking the target image as an image of a traffic violation vehicle as an example, the sample image is an image containing a real vehicle. After obtaining the basic key point detection model, feature points are extracted from the sample image through the basic key point detection model, and the key points are used as sample feature points for training the deep local feature point extraction model.

[0067] It should be noted that if Figure 3 As shown, the above-mentioned deep local feature point extraction model uses an encoding and decoding architecture to jointly detect and extract local feature points with a dual-branch network of feature detection and description, that is, the decoder consists of two decoders, which are used to output the position of the feature point and the vector of the feature point respectively.

[0068] The above scheme first performs simulation pre-training based on basic two-dimensional graphics to obtain a basic key point detection model, and then uses the basic key point model to perform key point detection on sample images containing real subjects, perform migration optimization of real vehicle data, and obtain pseudo-label information, namely, sample feature points. Finally, the deep local feature point extraction model is trained based on the pseudo-label information to obtain the final deep local feature point extraction model.

[0069] As an optional embodiment, sample feature points on the subject in the sample image are detected through a basic key point detection model, including: data augmenting the sample image by performing image transformation on the sample image; and detecting sample feature points on the subject in the sample image after data augmentation through a basic key point detection model.

[0070] Specifically, the above-mentioned image transformation is used to obtain multiple sample images after performing a specified form of transformation on a sample image, thereby playing a role in data augmentation. In an optional embodiment, the above-mentioned image transformation is used to represent the geometric transformation of the image, which may include: image translation transformation, image mirror transformation, image transposition transformation, image rotation transformation, image scaling, etc.

[0071] In order to train a more accurate deep local feature point extraction model, a large amount of pseudo-label information is required, but the number of sample images containing real subjects is limited, or it is difficult to obtain sample images containing real subjects. Therefore, the above scheme performs image transformation on the sample images for data augmentation, thereby greatly increasing the amount of sample data, and then being able to increase the amount of pseudo-label information, making the trained deep local feature point extraction model more accurate.

[0072] As an optional embodiment, the image transformation includes at least one of the following: translation, scaling, and scale flipping.

[0073] By transforming the image, multiple augmented images corresponding to the image can be obtained. In an optional embodiment, each sample image can be specified to be translated by two different distances, scaled by two different scales, and flipped once, so that five corresponding augmented images can be obtained for one sample image.

[0074] As an optional embodiment, based on the first local feature point information and the second local feature point information, a matching image that matches the target image is found from the candidate image, including: inputting the first local feature point information and the second local feature point information into a deep feature point matching model to obtain a similarity parameter between the candidate image output by the deep feature point matching model and the target image; and determining a matching image that matches the target image based on the similarity parameter.

[0075] In an optional embodiment, in combination Figure 3 As shown, the deep local feature point extraction model extracts local feature point sets from the target image and the candidate image respectively, obtains the vectors of the local feature point sets, and inputs this information into the deep feature point matching model to obtain the matching score output by the deep feature point matching model, which is used to represent the similarity between the target image and the candidate image. Then, based on the preset rules, according to the similarity between each candidate image and the target image, the candidate images with higher similarity are recalled, which can be the one with the highest similarity or multiple images ranked from high to low in similarity, so as to determine the matching image that matches the target image. The vehicle in the matching image is the same vehicle as the vehicle in the target image, and the identity of the vehicle in the target image can be determined based on the license plate information of the vehicle in the matching image.

[0076] As an optional embodiment, the deep feature point matching model includes: a corresponding point matching weight model and a basic matrix estimation model. The first local feature point information and the second local feature point information are input into the deep feature point matching model to obtain the similarity parameters between the candidate image and the target image output by the deep feature point matching model, including: predicting the matching probability between any two feature points in the first local feature point information and the second local feature point information through the corresponding point matching weight model; obtaining the affine transformation matrix between the candidate image and the target image according to the matching probability, the first local feature point information and the second local feature point information through the basic matrix estimation model; verifying the spatial relationship between the first local feature point in the candidate image and the second local feature point in the target image based on the affine transformation matrix to obtain the similarity parameters between the candidate image and the target image.

[0077] Since the candidate image and the target image have differences in shooting angle, light, etc., the result of directly matching feature points is not accurate, and the feature points must satisfy the spatial relationship between the candidate image and the target image. The above solution uses a deep feature point matching model to perform feature matching and spatial relationship verification based on the first local feature point information and the second local feature point information.

[0078] Specifically, the above-mentioned corresponding point weight matching model can be a modified residual network (Batch-Normalization is added to the residual network to make the network convergence better), which uses the corresponding point weight matching model to perform feature matching, predict the matching degree of any two points, and then determine the radial transformation matrix between the target image and the candidate image based on the matching degree. The spatial relationship between the feature points is verified according to the radial transformation matrix, that is, it is determined whether the feature points satisfy the spatial relationship defined by the radial transformation matrix. If so, the two feature points are considered to match.

[0079] The above-mentioned radial transformation matrix represents the spatial relationship between the candidate image and the target image. When determining the radial transformation matrix between the candidate image and the target object, the above-mentioned scheme also introduces the matching probability of any two points in the first local feature point information and the second local feature point information, thereby making the obtained radial transformation matrix more accurate.

[0080] After obtaining the affine transformation matrix between the candidate image and the target image, the spatial relationship between the first local feature point information and the second local feature point can be verified based on the spatial relationship between the candidate image and the target image described by the affine transformation matrix. During the verification, it is based on the assumption that if the candidate image and the target image match, the feature points of the two can be transformed based on the affine transformation matrix.

[0081] It should be noted that a joint learning framework can be used to jointly train the corresponding point matching weight model and the basic matrix estimation model using the improved weakly supervised classification loss, so that the final deep feature point matching network can accurately predict the matching degree of the two point sets.

[0082] It should also be noted that since the traditional matching method is executed in an iterative manner in the CPU, the execution efficiency is low. The above-mentioned scheme determines the similarity between the candidate image and the target image through a deep feature point matching model, which is executed in the GPU and does not require iterative operations. Therefore, it can solve the problems of RANSAC (Random Sample Consensus) mismatch and slow efficiency in traditional matching methods.

[0083] As an optional embodiment, the spatial relationship between the first local feature point in the candidate image and the second local feature point in the target image is verified based on the affine transformation matrix to obtain the similarity parameters between the candidate image and the target image, including: performing an affine transformation on the first local feature point according to the affine transformation matrix to obtain the transformation points corresponding to the first local feature point; obtaining the number of target first local feature points whose transformation points belong to the second local feature point; and determining the similarity parameters between the candidate image and the target image based on the number of target first local feature points and the number of first local feature points.

[0084] In the above scheme, the first local feature point is transformed through an affine transformation matrix to obtain a transformation point corresponding to the first local feature point. The meaning of the transformation point is that if the candidate image and the target image match, the transformation point corresponding to the feature point in a candidate matching object should belong to the second local feature point of the target image. Therefore, if it does not belong to the second local feature point, it is considered that at least this feature point does not satisfy the spatial relationship between the candidate image and the target image.

[0085] The above operation is performed on each first local feature point to obtain feature points that satisfy the spatial relationship between the candidate image and the target image, and feature points that do not satisfy the spatial relationship. The more feature points that satisfy the spatial relationship, the higher the similarity between the candidate image and the target image. Therefore, the ratio of the number of target first local feature points whose transformation points belong to the second local feature points to the number of all first local feature points can be used as the similarity between the candidate image and the target image.

[0086] As an optional embodiment, before searching for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information, the method further includes: displaying the target image and the second local feature points on the target image, wherein the information of the second local feature points constitutes the second local feature point information; displaying the candidate image and the first local feature points on the candidate image, wherein the information of the first local feature points constitutes the first local feature point information; receiving an adjustment instruction, wherein the adjustment instruction is used to adjust the area of ​​the first local feature point in the candidate image; and adjusting the position of the first local feature point in the candidate image according to the adjustment instruction.

[0087] In the above steps, the target image and the candidate image can be displayed on the human-computer interaction interface, and the first local feature point and the second local feature point can be displayed. Specifically, the second local feature point can be displayed by displaying the area where the second local feature point of the target image is located. Similarly, the first local feature point can be displayed by displaying the area where the first local feature point of the candidate image is located. The above adjustment instruction is used to adjust the area of ​​the first local feature point in the candidate image. During operation, the area where the first local feature point is located can be adjusted by comparing it with the area where the second local feature point in the target image is located. The object of adjustment can be the size, position, etc. of the area where the first local feature point is located.

[0088] Figure 4a is a schematic diagram of adjusting the first local feature point according to an embodiment of the present application, combined with Figure 4a As shown in the figure, the area where the second local feature point is located and the area where the first local feature point is located are framed by dotted lines in the target image and the candidate image. The two images above are the target image and the candidate image displayed in the initial state. It can be seen that the area where the first local feature point is located in the dotted line frame in the candidate image is larger than the area where the second local feature point is located in the target image. The user can adjust the size of the area framed by the dotted line in the candidate image to obtain Figure 4a As shown in the figure below, the area where the second local feature point framed in the adjusted target image and the candidate image is located is the same as the area where the first local feature point is located.

[0089] Figure 4b is another schematic diagram of adjusting the first local feature point according to an embodiment of the present application, combined with Figure 4bAs shown in the figure, the area where the second local feature point is located and the area where the first local feature point is located are framed by dotted lines in the target image and the candidate image. The two images above are the target image and the candidate image displayed in the initial state. It can be seen that the area where the first local feature point is located framed by the dotted line in the candidate image is deviated from the area where the second local feature point is located in the target image. The user can adjust the position of the area framed by the dotted line in the candidate image to obtain Figure 4b As shown in the figure below, the area where the second local feature point framed in the adjusted target image and the candidate image is located is the same as the area where the first local feature point is located.

[0090] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0092] Example 2

[0093] According to an embodiment of the present invention, there is also provided a Figure 5 The data processing method shown. Figure 5 is a flow chart of a data processing method according to Embodiment 2 of the present application. Figure 5 As shown, the method comprises the following steps:

[0094] Step S51, acquiring a target image of a target vehicle, wherein the target vehicle is a vehicle that causes a traffic violation, and the target image is an image of the target vehicle captured when the traffic violation occurs.

[0095] Specifically, the camera at the traffic checkpoint will take a picture of the passing vehicle. The above scheme uses the image of the illegal vehicle at the time of the violation as the target image for matching, and is used to find a vehicle matching the illegal vehicle through the target image when the license plate of the illegal vehicle is not clearly photographed and the identity of the illegal vehicle cannot be determined, thereby determining the identity information of the illegal vehicle.

[0096] Step S53, searching for candidate vehicles corresponding to the target vehicle from a vehicle image library according to the target image.

[0097] Specifically, the image library may be a database of a large number of images containing vehicles, and in the image library, vehicles in the acquired images may be extracted and then stored. The candidate image may be an image obtained by preliminary screening from the image library based on one or more global features in the target image. The global features may include any one or more of the following: color, model, brand, etc.

[0098] In an optional embodiment, images of vehicles captured by cameras at traffic checkpoints throughout the city are acquired in real time, and the vehicles in the images are extracted and stored in an image library. When a traffic checkpoint detects that there is an illegal vehicle, but the vehicle license plate is difficult to accurately identify due to angle, light or human factors, the image captured when the illegal vehicle violates the traffic rules is used as the target image, and the corresponding candidate image is searched in the image library based on the global features of the illegal vehicle.

[0099] Through the above steps, vehicle images similar to the vehicle in the target image can be screened out from the massive vehicle images stored in the image library, thereby excluding most vehicles. After the candidate images are obtained, further searches are performed in the candidate images.

[0100] Step S55, obtaining the first local feature point information of the candidate vehicle in the candidate vehicle image.

[0101] Specifically, the candidate image has multiple local feature points on the vehicle, and the first local feature point information is a set of vectors of multiple local feature points on the candidate image. Local feature points are used to describe local features, which are features extracted from local areas of the image, including edges, corners, lines, curves, and areas with special attributes. Compared with global image features such as line features, texture features, and structural features, local features of an image have the characteristics of being abundant in number, having low correlation between features, and not affecting the detection and matching of other features due to the disappearance of some features in the case of occlusion. Local features also have the characteristics of high feature detection repetition rate and fast speed, robust feature description to image transformations such as illumination, rotation, and viewpoint changes, low dimensionality of feature descriptors, and easy to achieve fast matching.

[0102] Step S57, obtaining second local feature point information of the target image.

[0103] In an optional embodiment, taking the target image as an image of a traffic violation vehicle as an example, the local features of the target image can still be extracted by a preset local feature point extraction model to obtain the second local feature point information of the target image.

[0104] Step S59: searching for a matching vehicle that matches the target vehicle from the candidate vehicles based on the first local feature point information and the second local feature point information.

[0105] In the above scheme, after obtaining the first local feature point information and the second local feature point information, the similarity between the two can be obtained, and the similarity is used to represent the similarity between the candidate image and the target image, so as to determine the matching image that best matches the target image according to the similarity.

[0106] In an optional embodiment, still taking the image of the illegal vehicle as the target image as an example, when the vehicle violates the traffic rules, it is difficult to accurately identify the vehicle license plate due to angle, light or human factors, and thus the identity of the vehicle cannot be identified. However, the vehicle will also leave images at other traffic checkpoints, and the images left at other traffic checkpoints may contain the vehicle license plate. In this case, by matching the local feature point information, the matching image that best matches the target image is found from the candidate images, and it is considered that the subject in the matching image is the above-mentioned illegal vehicle, and the identity of the vehicle can be determined based on the license plate in the matching image.

[0107] This solution uses image search to authenticate the identity of vehicles, which can significantly improve the search accuracy of the vehicle image search system when there are many vehicles of the same brand, providing a powerful means for the control and inspection of illegal vehicles in traffic scenes.

[0108] The above embodiment of the present application obtains a target image of a target vehicle, wherein the target vehicle is a vehicle that generates a traffic violation, and the target image is an image of the target vehicle captured when the traffic violation occurs; searches for a candidate vehicle corresponding to the target vehicle from a vehicle image library based on the target image; obtains first local feature point information of the candidate vehicle in the candidate vehicle image; obtains second local feature point information of the target image; and searches for a matching vehicle that matches the target vehicle from the candidate vehicles based on the first local feature point information and the second local feature point information. The above scheme performs matching based on the local feature information of the candidate image and the local feature information of the target image, based on the preliminary extraction of the candidate image of the target image from the image library, so that the detailed information of the image can be perceived, and the accuracy of image matching can be improved, thereby solving the technical problem of low accuracy in image matching in the prior art.

[0109] This embodiment may also include other steps in Embodiment 1 if there is no conflict, which will not be described in detail here.

[0110] Example 3

[0111] According to an embodiment of the present invention, there is also provided a Figure 6 The image matching system is shown. Figure 6 is a schematic diagram of an image matching system according to Embodiment 3 of the present application. Figure 6 As shown, the system includes:

[0112] The image acquisition device 60 is used to acquire a target image.

[0113] Processor 62 is used to obtain a target image and obtain second local feature point information of the target image, and search for a candidate image corresponding to the target image from an image library; obtain first local feature point information of the candidate image, and based on the first local feature point information and the second local feature point information, search for a matching image that matches the target image from the candidate image, wherein the first local feature point information includes a local feature point vector of a subject in the candidate image, and the second local feature point information includes a local feature point vector of a subject in the target image.

[0114] Specifically, the target image may be an image containing a subject to be searched, and the image library may be a database of a large number of images containing subjects, and in the image library, the subject in the acquired image may be extracted and then stored. The candidate image may be an image obtained by preliminary screening from the image library based on one or more global features in the target image. The global features may include any one or more of the following: color, model, brand, etc.

[0115] By searching for candidate images corresponding to the target image, vehicle images similar to the vehicle in the target image can be screened out from the massive vehicle images stored in the image library, thereby excluding most vehicles. After the candidate images are obtained, further searches are performed in the candidate images.

[0116] The subject of the candidate image has multiple local feature points, and the first local feature point information is a set of vectors of multiple local feature points on the subject of the candidate image. Local feature points are used to describe local features, which are features extracted from local areas of the image, including edges, corners, lines, curves, and areas with special attributes. Compared with global image features such as line features, texture features, and structural features, local features of an image have the characteristics of being abundant in number, having low correlation between features, and not affecting the detection and matching of other features due to the disappearance of some features in the case of occlusion. Local features also have the characteristics of high feature detection repetition rate and fast speed, feature description is robust to image transformations such as illumination, rotation, and viewpoint changes, feature descriptor has low dimension, and is easy to achieve fast matching.

[0117] In the above scheme, after obtaining the first local feature point information and the second local feature point information, the similarity between the two can be obtained, and the similarity is used to represent the similarity between the candidate image and the target image, so as to determine the matching image that best matches the target image according to the similarity.

[0118] In an optional embodiment, still taking the image of the illegal vehicle as the target image as an example, when the vehicle violates the traffic rules, it is difficult to accurately identify the vehicle license plate due to angle, light or human factors, and thus the identity of the vehicle cannot be identified. However, the vehicle will also leave images at other traffic checkpoints, and the images left at other traffic checkpoints may contain the vehicle license plate. In this case, by matching the local feature point information, the matching image that best matches the target image is found from the candidate images, and it is considered that the subject in the matching image is the above-mentioned illegal vehicle, and the identity of the vehicle can be determined based on the license plate in the matching image.

[0119] This solution uses image search to authenticate the identity of vehicles, which can significantly improve the search accuracy of the vehicle image search system when there are many vehicles of the same brand, providing a powerful means for the control and inspection of illegal vehicles in traffic scenes.

[0120] As an optional embodiment, the above system also includes: a human-computer interaction device, which communicates with the processor, is used to receive a selection instruction, and obtains the local feature point information of the area to be matched through a preset deep local feature point extraction model, and determines that the local feature point information of the area to be matched is the second local feature point information, wherein the selection instruction is used to select the area to be matched in the target image.

[0121] Specifically, the above-mentioned preset deep local feature point extraction model may be the same as the model used to extract the first local feature point information.

[0122] In an optional embodiment, the above selection instruction can be issued by the user to select the area to be matched in the target image. For example, if there is an area with obvious features in the main body of the target image, the user can select this area, so as to use the deep local feature point extraction model to extract local feature point information in the selected area, thereby making the extracted local feature point information more targeted. Still taking the target image as an image of an illegal vehicle as an example, if the target image captures a banner hung on the front of the illegal vehicle, the area where the banner is located can be selected as the area to be matched.

[0123] The processor in this embodiment may also execute other steps in Embodiment 1 if there is no conflict, which will not be described in detail here.

[0124] Example 4

[0125] According to an embodiment of the present invention, there is also provided a data processing device for implementing the data processing method of the above embodiment 1. Figure 7 is a schematic diagram of a data processing device according to Embodiment 4 of the present application, such as Figure 7 As shown, the device 700 includes:

[0126] A first acquisition module 702 is used to acquire a target image, wherein the target image includes a main body part and a non-main body part;

[0127] A second acquisition module 704 is used to acquire second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of a subject in the target image;

[0128] The first search module 706 is used to search for a candidate image corresponding to the target image from the image library.

[0129] The third acquisition module 708 is used to acquire first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of the subject in the candidate image.

[0130] The second search module 7010 is used to search for a matching image that matches the target image from the candidate images based on the first local feature point information and the second local feature point information.

[0131] It should be noted that the first acquisition module 702, the second acquisition module 704, the first search module 706, the third acquisition module 708 and the second search module 7010 correspond to steps S21 to S29 in Example 1, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment 1. It should be noted that the above-mentioned modules as part of the apparatus can be run in the computing device 10 provided in the embodiment 1.

[0132] As an optional embodiment, the first search module includes: an extraction submodule, used to extract global feature information of the target image; and a search submodule, used to search for candidate images in the image library based on the global feature information.

[0133] As an optional embodiment, the third acquisition module includes: a first acquisition submodule, configured to acquire first local feature point information of the candidate image by using a preset deep local feature point extraction model.

[0134] As an optional embodiment, the second acquisition module includes: a receiving submodule, used to receive a selection instruction, wherein the selection instruction is used to select a to-be-matched area in the target image; a second acquisition submodule, used to obtain local feature point information of the to-be-matched area through a preset deep local feature point extraction model, and determine the local feature point information of the to-be-matched area as the second local feature point information.

[0135] As an optional embodiment, the above-mentioned device also includes: a third acquisition module, used to obtain a deep local feature point extraction model, the third acquisition module includes: a first training submodule, used to train based on the key points in a preset two-dimensional graphic to obtain a basic key point detection model; a detection submodule, used to detect sample feature points on the subject in the sample image through the basic key point detection model; a second training submodule, used to use the sample feature points as sample data to train a preset initial network model to obtain a deep local feature point extraction model, wherein the preset initial network model includes an encoder and two decoders, and the two decoders are respectively used to predict the position and vector of the feature points.

[0136] As an optional embodiment, the detection submodule includes: a transformation unit, which is used to perform data augmentation on the sample image by performing image transformation on the sample image; and a detection unit, which is used to detect sample feature points on the subject in the sample image after data augmentation through a basic key point detection model.

[0137] As an optional embodiment, the image transformation includes at least one of the following: translation, scaling, and scale flipping.

[0138] As an optional embodiment, the second search module includes: an input submodule, used to input the first local feature point information and the second local feature point information into the deep feature point matching model to obtain the similarity parameters between the candidate image output by the deep feature point matching model and the target image; a determination submodule, used to determine the matching image that matches the target image based on the similarity parameters.

[0139] As an optional embodiment, the deep feature point matching model includes: a corresponding point matching weight model and a basic matrix estimation model, and the input submodule includes: a prediction unit, used to predict the matching probability between any two feature points in the first local feature point information and the second local feature point information through the corresponding point matching weight model; an acquisition unit, used to acquire the affine transformation matrix between the candidate image and the target image according to the matching probability, the first local feature point information and the second local feature point information through the basic matrix estimation model; a verification unit, used to verify the spatial relationship between the first local feature point in the candidate image and the second local feature point in the target image based on the affine transformation matrix, and obtain the similarity parameter between the candidate image and the target image.

[0140] As an optional embodiment, the verification unit includes: a transformation subunit, used to perform affine transformation on the first local feature point according to an affine transformation matrix to obtain the transformation point corresponding to the first local feature point; an acquisition subunit, used to obtain the number of target first local feature points whose transformation points belong to the second local feature point; and a determination subunit, used to determine the similarity parameter between the candidate image and the target image based on the number of target first local feature points and the number of first local feature points.

[0141] As an optional embodiment, the above-mentioned device also includes: a first display module, used to display the target image and the second local feature points on the target image before searching for a matching image matching the target image from the candidate image based on the first local feature point information and the second local feature point information, wherein the information of the second local feature points constitutes the second local feature point information; a second display module, used to display the candidate image and the first local feature points on the candidate image, wherein the information of the first local feature points constitutes the first local feature point information; a receiving module, used to receive an adjustment instruction, wherein the adjustment instruction is used to adjust the area of ​​the first local feature point in the candidate image; and an adjustment module, used to adjust the position of the first local feature point in the candidate image according to the adjustment instruction.

[0142] Example 5

[0143] According to an embodiment of the present invention, there is also provided a data processing device for implementing the data processing method of the above embodiment 2. Figure 8is a schematic diagram of a data processing device according to Embodiment 5 of the present application, such as Figure 8 As shown, the device 700 includes:

[0144] The first acquisition module 802 is used to acquire a target image of a target vehicle, wherein the target vehicle is a vehicle that causes a traffic violation, and the target image is an image captured when the traffic violation occurs.

[0145] The first search module 804 is used to search for candidate vehicles corresponding to the target vehicle from the vehicle image library according to the target image.

[0146] The second acquisition module 806 is used to acquire first local feature point information of the candidate vehicle in the candidate vehicle image.

[0147] The third acquisition module 808 is used to acquire second local feature point information of the target image.

[0148] The second search module 8010 is used to search for a matching vehicle that matches the target vehicle from the candidate vehicles based on the first local feature point information and the second local feature point information.

[0149] It should be noted that the first search module 804, the second acquisition module 806, the second acquisition module 806, the third acquisition module 808 and the second search module 8010 correspond to steps S51 to S59 in Example 2, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment 1. It should be noted that the above-mentioned modules as part of the apparatus can be run in the computing device 10 provided in the embodiment 1.

[0150] Example 6

[0151] The embodiment of the present invention may provide a computing device, which may be any computing device in a computing device group. Optionally, in this embodiment, the computing device may also be replaced by a terminal device such as a mobile terminal.

[0152] Optionally, in this embodiment, the computing device may be located in at least one network device among a plurality of network devices of a computer network.

[0153] In this embodiment, the above-mentioned computing device can execute the program code of the following steps in the vulnerability detection method of the application: obtaining a target image, wherein the target image includes a main part and a non-main part; obtaining second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of the main body in the target image; searching for a candidate image corresponding to the target image from an image library; obtaining first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of the main body in the candidate image; based on the first local feature point information and the second local feature point information, searching for a matching image that matches the target image from the candidate image.

[0154] Optionally, Fig. 9 is a structural block diagram of a computing device according to Embodiment 6 of the present application. Fig. 9 As shown, the computing device A may include: one or more (only one is shown in the figure) processors 902 , a memory 906 , and a peripheral interface 908 .

[0155] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the security vulnerability detection method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the detection method of the above-mentioned system vulnerability attack is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0156] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a target image, wherein the target image includes a main part and a non-main part; obtain second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of the main body in the target image; search for a candidate image corresponding to the target image from an image library; obtain first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of the main body in the candidate image; based on the first local feature point information and the second local feature point information, search for a matching image that matches the target image from the candidate image.

[0157] Optionally, the processor may also execute program codes of the following steps: extracting global feature information of the target image; and searching for candidate images in an image library based on the global feature information.

[0158] Optionally, the processor may also execute a program code of the following steps: obtaining first local feature point information of the candidate image by using a preset deep local feature point extraction model.

[0159] Optionally, the processor may also execute the program code of the following steps: receiving a selection instruction, wherein the selection instruction is used to select an area to be matched in the target image; obtaining local feature point information of the area to be matched through a preset deep local feature point extraction model, and determining that the local feature point information of the area to be matched is the second local feature point information.

[0160] Optionally, the processor may also execute the program code of the following steps: obtaining a deep local feature point extraction model, wherein obtaining the deep local feature point extraction model includes: training based on key points in a preset two-dimensional graphic to obtain a basic key point detection model; detecting sample feature points on the subject in the sample image through the basic key point detection model; using the sample feature points as sample data, training a preset initial network model to obtain a deep local feature point extraction model, wherein the preset initial network model includes an encoder and two decoders, and the two decoders are respectively used to predict the position and vector of the feature points.

[0161] Optionally, the processor may also execute program codes of the following steps: data augmenting the sample image by performing image transformation on the sample image; and detecting sample feature points on the subject in the sample image after data augmentation by using a basic key point detection model.

[0162] Optionally, the image transformation includes at least one of the following: translation, scaling, and scale flipping.

[0163] Optionally, the processor may also execute the program code of the following steps: based on the first local feature point information and the second local feature point information, input the first local feature point information and the second local feature point information into a deep feature point matching model to obtain similarity parameters between the candidate image output by the deep feature point matching model and the target image; and determine a matching image that matches the target image based on the similarity parameters.

[0164] Optionally, the deep feature point matching model includes: a corresponding point matching weight model and a basic matrix estimation model, and the above-mentioned processor can also execute the program code of the following steps: predicting the matching probability between any two feature points in the first local feature point information and the second local feature point information through the corresponding point matching weight model; obtaining the affine transformation matrix between the candidate image and the target image according to the matching probability, the first local feature point information and the second local feature point information through the basic matrix estimation model; verifying the spatial relationship between the first local feature point in the candidate image and the second local feature point in the target image based on the affine transformation matrix, and obtaining the similarity parameters between the candidate image and the target image.

[0165] Optionally, the processor may also execute the program code of the following steps: performing an affine transformation on the first local feature point according to an affine transformation matrix to obtain the transformation point corresponding to the first local feature point; obtaining the number of target first local feature points whose transformation points belong to the second local feature point; and determining the similarity parameter between the candidate image and the target image based on the number of target first local feature points and the number of first local feature points.

[0166] Optionally, the processor may also execute program code for the following steps: displaying the target image and the second local feature points on the target image before searching for a matching image that matches the target image from the candidate images based on the first local feature point information and the second local feature point information, wherein the information of the second local feature points constitutes the second local feature point information; displaying the candidate image and the first local feature points on the candidate image, wherein the information of the first local feature points constitutes the first local feature point information; receiving an adjustment instruction, wherein the adjustment instruction is used to adjust the area of ​​the first local feature point in the candidate image; and adjusting the position of the first local feature point in the candidate image according to the adjustment instruction.

[0167] According to an embodiment of the present invention, a data processing method is provided. A target image is obtained, wherein the target image includes a main body part and a non-main body part; second local feature point information of the target image is obtained, wherein the second local feature point information includes a local feature point vector of the main body in the target image; a candidate image corresponding to the target image is searched from an image library; first local feature point information of the candidate image is obtained, wherein the first local feature point information includes a local feature point vector of the main body in the candidate image; based on the first local feature point information and the second local feature point information, a matching image that matches the target image is searched from the candidate image. The above scheme performs matching based on the local feature information of the candidate image and the local feature information of the target image on the basis of preliminarily extracting the candidate image of the target image from the image library, thereby being able to perceive the detailed information of the image, thereby being able to improve the accuracy of image matching, and solving the technical problem of low accuracy in image matching in the prior art.

[0168] It can be understood by those skilled in the art that Fig. 9 The structure shown is for illustration only, and the computing device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig. 9 The structure of the electronic device is not limited. For example, the computing device 80 may also include Fig. 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig. 9 Different configurations shown.

[0169] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0170] Example 7

[0171] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0172] Optionally, in this embodiment, the above storage medium may be located in any one of the computing devices in the computing device group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0173] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: obtaining a target image, wherein the target image includes a main body part and a non-main body part; obtaining second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of the main body in the target image; searching for a candidate image corresponding to the target image from an image library; obtaining first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of the main body in the candidate image; and searching for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information.

[0174] Example 8

[0175] According to an embodiment of the present invention, there is also provided a Fig.10 The data processing method shown. Fig.10 is a flow chart of a data processing method according to Embodiment 8 of the present application. Fig.10 As shown, the method comprises the following steps:

[0176] Step S101, acquiring a target image of a target vehicle.

[0177] Specifically, the target vehicle may be a vehicle whose running trajectory is to be determined, such as a vehicle violating traffic rules, and the target image of the target vehicle may be any image taken by a camera at a traffic checkpoint.

[0178] Step S103: Acquire local feature point information of the target image, wherein the local feature point information includes a local feature point vector of the target vehicle.

[0179] Specifically, the local feature point information of the target image can be the local feature point information on the target vehicle in the target image, and the local feature point information is a collection of vectors of multiple local feature points on the target vehicle. Local feature points are used to describe local features, which are features extracted from local areas of the image, including edges, corners, lines, curves, and areas with special attributes. Compared with global image features such as line features, texture features, and structural features, local features of an image have the characteristics of being abundant in number, having low correlation between features, and not affecting the detection and matching of other features due to the disappearance of some features in the case of occlusion. Local features also have the characteristics of high feature detection repetition rate and fast speed, robust feature description to image transformations such as illumination, rotation, and viewpoint changes, low dimensionality of feature descriptors, and easy to achieve fast matching.

[0180] In an optional embodiment, local features on the target vehicle in the target image may be extracted using a preset local feature point extraction model to obtain local feature point information of the target image.

[0181] Step S105 , searching for a matching image that matches the target image from a gallery according to the local feature point information of the target image, wherein the gallery includes images captured by image capture devices at different locations.

[0182] Specifically, the image acquisition devices at different positions may be cameras arranged at different traffic checkpoints, and the matching images searched from the images by the above scheme are images including the target vehicle acquired by different cameras.

[0183] In an optional embodiment, first, according to the global feature information of the target vehicle (for example, the color, model, etc. of the vehicle), a candidate image corresponding to the target image can be searched from the image library, and then the local feature point information of the candidate image can be obtained, and finally, based on the local feature point information of the target image and the local feature point information of the candidate image, a matching image matching the target image can be searched from the candidate image. This process can be implemented by applying the data processing method in Example 1, which will not be described in detail here.

[0184] Step S105, determining the running track of the target vehicle according to the position of the image acquisition device and the time when the matching image is acquired.

[0185] When the target vehicle is in driving state, it will pass through multiple different traffic checkpoints, so the cameras at different traffic checkpoints will capture images of the target vehicle at different times, i.e. the matching images mentioned above. Therefore, based on the location of the camera and the time when the camera captured the matching image, the location of the target vehicle at the time of shooting can be determined, and then the running track of the target vehicle can be determined.

[0186] In an optional embodiment, after obtaining all matching images of the target vehicle within a preset time and the acquisition time of the matching images by the camera, the position of the camera of the acquired matching images is first obtained, and then the matching images are sorted according to the order of the time when the camera captured the matching images. Finally, the position of the camera is connected according to the sorting results to obtain the running trajectory of the target vehicle.

[0187] Through the above scheme, the running trajectory of the vehicle can be depicted based on the matching of the target image of the target vehicle, thereby achieving accurate tracking of the vehicle.

[0188] Example 9

[0189] According to an embodiment of the present invention, there is also provided a data processing device for implementing the data processing method of the above-mentioned embodiment 8. Fig.11 is a schematic diagram of a data processing device according to Embodiment 9 of the present application, such as Fig.11 As shown, the device 1100 includes:

[0190] The first acquisition module 1102 is used to acquire a target image of a target vehicle.

[0191] The second acquisition module 1104 is used to acquire local feature point information of the target image, wherein the local feature point information includes a local feature point vector of the target vehicle.

[0192] The search module 1106 is used to search for a matching image that matches the target image from a gallery according to the local feature point information of the target image, wherein the gallery includes images collected by image collection devices at different locations.

[0193] The determination module 1108 is used to determine the running track of the target vehicle according to the position of the image acquisition device and the time when the matching image is acquired.

[0194] It should be noted that the first acquisition module 1102, the second acquisition module 1104, the search module 1106 and the search module 1106 correspond to steps S101 to S107 in Example 8, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment 1. It should be noted that the above-mentioned modules as part of the apparatus can be run in the computing device 10 provided in the embodiment 1.

[0195] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0196] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0197] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0198] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0201] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Acquire a target image, wherein the target image includes a main body part and a non-main body part; Acquire second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of a subject in the target image; Searching for a candidate image corresponding to the target image from an image library; Acquire first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of a subject in the candidate image; Based on the first local feature point information and the second local feature point information, searching for a matching image that matches the target image from the candidate images; wherein the feature points of the target image and the feature points in the matching image satisfy the spatial relationship between the candidate image and the target image; Among them, the method of searching for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information includes: predicting the matching probability between any two feature points in the first local feature point information and the second local feature point information through a corresponding point matching weight model; obtaining an affine transformation matrix between the candidate image and the target image according to the matching probability, the first local feature point information and the second local feature point information through a basic matrix estimation model; verifying the spatial relationship between the first local feature point in the candidate image and the second local feature point in the target image based on the affine transformation matrix to obtain a similarity parameter between the candidate image and the target image; and determining a matching image that matches the target image based on the similarity parameter.

2. The method according to claim 1, characterized in that Searching for a candidate image corresponding to the target image from an image library includes: Extracting global feature information of the target image; The candidate image is searched in the image library based on the global feature information.

3. The method according to claim 1, characterized in that Acquiring first local feature point information of the candidate image, including: The first local feature point information of the candidate image is obtained through a preset deep local feature point extraction model.

4. The method according to claim 1, characterized in that: Acquiring second local feature point information of the target image, including: Receiving a selection instruction, wherein the selection instruction is used to select a to-be-matched area in the target image; The local feature point information of the to-be-matched area is acquired through a preset deep local feature point extraction model, and the local feature point information of the to-be-matched area is determined to be the second local feature point information.

5. The method according to claim 3, characterized in that: The method further includes: obtaining the deep local feature point extraction model, and obtaining the deep local feature point extraction model includes: Based on the key points in the preset two-dimensional graphics, training is performed to obtain a basic key point detection model; Detecting sample feature points on a subject in a sample image by using the basic key point detection model; The sample feature points are used as sample data to train a preset initial network model to obtain the deep local feature point extraction model, wherein the preset initial network model includes an encoder and two decoders, and the two decoders are respectively used to predict the position and vector of the feature points.

6. The method according to claim 5, characterized in that: Detecting sample feature points on a subject in a sample image by using the basic key point detection model includes: Performing data augmentation on the sample image by performing image transformation on the sample image; The basic key point detection model is used to detect sample feature points on the subject in the sample image after data augmentation.

7. The method according to claim 6, characterized in that The image transformation includes at least one of the following: translation, scaling and scale flipping.

8. The method according to claim 1, characterized in that Verifying the spatial relationship between the first local feature point in the candidate image and the second local feature point in the target image based on the affine transformation matrix to obtain a similarity parameter between the candidate image and the target image includes: Performing an affine transformation on the first local feature point according to the affine transformation matrix to obtain a transformation point corresponding to the first local feature point; Obtain the number of target first local feature points whose transformation points belong to the second local feature points; A similarity parameter between the candidate image and the target image is determined according to the number of the target first local feature points and the number of the first local feature points.

9. The method according to claim 1, characterized in that: Before searching for a matching image that matches the target image from the candidate images based on the first local feature point information and the second local feature point information, the method further includes: Displaying the target image and second local feature points on the target image, wherein information of the second local feature points constitutes the second local feature point information; Display the candidate image and the first local feature point on the candidate image, wherein information of the first local feature point constitutes the first local feature point information; receiving an adjustment instruction, wherein the adjustment instruction is used to adjust the area of ​​the first local feature point in the candidate image; The position of the first local feature point in the candidate image is adjusted according to the adjustment instruction.

10. A data processing method, characterized in that: include: Acquire a target image of a target vehicle, wherein the target vehicle is a vehicle that causes a traffic violation, and the target image is an image of the target vehicle captured when the traffic violation occurs; Searching for candidate vehicles corresponding to the target vehicle from a vehicle image library according to the target image; Acquire first local feature point information of the candidate vehicle in the candidate vehicle image; Acquire second local feature point information of the target image; Based on the first local feature point information and the second local feature point information, searching for a matching vehicle that matches the target vehicle from the candidate vehicles; wherein the feature points in the target image corresponding to the target vehicle and the feature points in the matching image corresponding to the matching vehicle satisfy the spatial relationship between the candidate vehicle image and the target image; Among them, based on the first local feature point information and the second local feature point information, searching for a matching vehicle that matches the target vehicle from the candidate vehicles includes: predicting the matching probability between any two feature points in the first local feature point information and the second local feature point information through a corresponding point matching weight model; obtaining an affine transformation matrix between the candidate vehicle image and the target image according to the matching probability, the first local feature point information and the second local feature point information through a basic matrix estimation model; verifying the spatial relationship between the first local feature point in the candidate vehicle image and the second local feature point in the target image based on the affine transformation matrix to obtain a similarity parameter between the candidate vehicle image and the target image; and determining a matching image that matches the target image based on the similarity parameter.

11. A data processing device, characterized in that: include: A first acquisition module, used for acquiring a target image, wherein the target image includes a main body part and a non-main body part; A second acquisition module, configured to acquire second local feature point information of the target image, wherein the second local feature point information includes a local feature point vector of a subject in the target image; A first search module, used to search for a candidate image corresponding to the target image from an image library; A third acquisition module, configured to acquire first local feature point information of the candidate image, wherein the first local feature point information includes a local feature point vector of a subject in the candidate image; A second search module, configured to search for a matching image that matches the target image from the candidate images based on the first local feature point information and the second local feature point information; Wherein, the feature points in the target image and the feature points in the matching image satisfy the spatial relationship between the candidate image and the target image; Among them, the second search module is also used to find a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information through the following steps: predicting the matching probability between any two feature points in the first local feature point information and the second local feature point information through a corresponding point matching weight model; obtaining an affine transformation matrix between the candidate image and the target image according to the matching probability, the first local feature point information and the second local feature point information through a basic matrix estimation model; verifying the spatial relationship between the first local feature point in the candidate image and the second local feature point in the target image based on the affine transformation matrix to obtain a similarity parameter between the candidate image and the target image; and determining a matching image that matches the target image based on the similarity parameter.

12. A data processing system, characterized in that: include: An image acquisition device, used for acquiring a target image; A processor, configured to obtain a target image and obtain second local feature point information of the target image, search for a candidate image corresponding to the target image from an image library, obtain first local feature point information of the candidate image, and search for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information, wherein the target image includes a main body part and a non-main body part, the first local feature point information includes a local feature point vector of the main body in the candidate image, and the second local feature point information includes a local feature point vector of the main body in the target image; wherein the feature points in the target image and the feature points in the matching image satisfy a spatial relationship between the candidate image and the target image; Among them, the method of searching for a matching image that matches the target image from the candidate image based on the first local feature point information and the second local feature point information includes: predicting the matching probability between any two feature points in the first local feature point information and the second local feature point information through a corresponding point matching weight model; obtaining an affine transformation matrix between the candidate image and the target image according to the matching probability, the first local feature point information and the second local feature point information through a basic matrix estimation model; verifying the spatial relationship between the first local feature point in the candidate image and the second local feature point in the target image based on the affine transformation matrix to obtain a similarity parameter between the candidate image and the target image; and determining a matching image that matches the target image based on the similarity parameter.

13. The system according to claim 12, characterized in that The system further comprises: A human-computer interaction device communicates with the processor, is used to receive a selection instruction, and obtains local feature point information of the area to be matched through a preset deep local feature point extraction model, and determines that the local feature point information of the area to be matched is the second local feature point information, wherein the selection instruction is used to select the area to be matched in the target image.

14. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 10.

15. A processor, characterized in that: The processor is used to run a program, wherein the program executes the data processing method according to any one of claims 1 to 10 when running.

16. A data processing method, characterized in that: include: Acquire a target image of a target vehicle; Acquire local feature point information of the target image, wherein the local feature point information includes a local feature point vector of the target vehicle; According to the local feature point information of the target image, a matching image matching the target image is searched from a library, wherein the library includes images collected by image collection devices at different positions; Determine the running track of the target vehicle according to the position of the image acquisition device and the time when the matching image is acquired; wherein the feature points in the target image and the feature points in the matching image satisfy the spatial relationship between the candidate image and the target image; Among them, according to the local feature point information of the target image, a matching image matching the target image is searched from a gallery, including: predicting the matching probability between any two feature points in the local feature point information of the target image and the local feature point information of the image in the gallery through a corresponding point matching weight model; obtaining the affine transformation matrix between the image in the gallery and the target image according to the matching probability, the local feature point information of the target image and the local feature point information of the image in the gallery through a basic matrix estimation model; verifying the spatial relationship between the local feature points in the image in the gallery and the local feature points in the target image based on the affine transformation matrix to obtain the similarity parameters between the image in the gallery and the target image; and determining the matching image matching the target image based on the similarity parameters.

Images