Image processing method and device, equipment and storage medium
By using image processing technology in the image plagiarism checking system, the points of interest and surrounding image features of the image plagiarism checking system are extracted and matched, and the problems of low efficiency, low accuracy and high labor cost in the prior art are solved. , achieving more efficient and accurate picture plagiarism check.
Patent Information
- Application Number
- CN202411461953.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-02-18
AI Technical Summary
The existing technology for maintaining image plagiarism checking is low efficiency, low accuracy and high labor costs, making it difficult to effectively identify and process the behavior of maintaining personnel in false orders through image processing software.
Using image processing method, by obtaining the current generation of dimensional images and inputting them into the plagiarism check model, the features of the point of interest and the surrounding image features are extracted, and the image plagiarism check is automatically performed by combining the image feature matching network.
It improves the efficiency and accuracy of picture plagiarism checking, reduces labor costs, and can more effectively identify and deal with false order replies.
Smart Images
Figure CN119478458A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an image processing method, device, equipment and storage medium. Background Art
[0002] The duplicate check of maintenance pictures is mainly used in the audit of maintenance work photos. It matches the newly uploaded maintenance work photos with the historical work photos to determine whether the maintenance personnel use the historical work photos to falsely return the receipt. If the newly added work photos fail the duplicate check, it means that the maintenance personnel may have processed the historical photos by cropping, filtering, rotating the pictures, etc., and uploaded them as proof materials for the new work order, but in fact did not perform the relevant work, which is a fraudulent behavior.
[0003] At present, image maintenance mainly relies on manual spot checks and audits. Different users have different standards for checking for duplicate content. The efficiency of image duplication checking is low, the accuracy is low, and the labor cost is high. Summary of the invention
[0004] The present disclosure provides an image processing method, apparatus, device and storage medium to at least solve the problems of low efficiency, low accuracy and high labor cost in existing image duplication checking.
[0005] The technical solution of the present disclosure is as follows:
[0006] The present disclosure provides an image processing method, including:
[0007] Get the current maintenance image;
[0008] Input the current proxy image into a duplicate checking model to obtain a first point of interest feature of the current proxy image and a first surrounding image feature of each first point of interest;
[0009] Input the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, and the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points into an image feature matching network to obtain a matching result between the current proxy image and the target proxy image;
[0010] Among them, the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model.
[0011] Optionally, the image feature matching network includes: an attention graph neural network and an optimal matching layer, and the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points are input into the image feature matching network to obtain a matching result between the current proxy image and the target proxy image, including:
[0012] Inside the image feature matching network, the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points are input into the attention graph neural network to obtain a first matching descriptor of the current proxy image and a second matching descriptor of the target proxy image;
[0013] The first matching descriptor and the second matching descriptor are input into the optimal matching layer to obtain a matching result between the current proxy image and the target proxy image.
[0014] Optionally, the step of inputting the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points into the attention graph neural network to obtain a first matching descriptor of the current proxy image and a second matching descriptor of the target proxy image includes:
[0015] Input the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, and the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points into a feature point encoder respectively, to obtain a first feature vector of the current proxy image and a second feature vector of the target proxy image;
[0016] The first feature vector and the second feature vector are linearly projected respectively to obtain the first matching descriptor of the current proxy image and the second matching descriptor of the target proxy image.
[0017] Optionally, the inputting the first matching descriptor and the second matching descriptor into the optimal matching layer to obtain a matching result between the current proxy image and the target proxy image includes:
[0018] Taking the first matching descriptor and the second matching descriptor as an inner product to obtain a distribution matrix, wherein the distribution matrix is used to represent the similarity between the first matching descriptor and the second matching descriptor;
[0019] Counting the number of element values of the allocation matrix that are greater than a confidence threshold;
[0020] The matching result between the current proxy image and the target proxy image is determined according to the number of element values of the allocation matrix that are greater than the confidence threshold.
[0021] Optionally, determining the matching result between the current proxy image and the target proxy image according to the number of element values of the allocation matrix that are greater than a confidence threshold includes:
[0022] If the number is greater than or equal to the number threshold, it is determined that the current proxy image matches the target proxy image successfully;
[0023] When the number is less than the number threshold, it is determined that the current proxy image and the target proxy image are not successfully matched.
[0024] Optionally, the method further comprises:
[0025] If there is a proxy image in the image feature library that matches the current proxy image successfully, a re-maintenance message is generated;
[0026] The re-maintenance message is sent to the maintenance terminal so that the maintenance user can view the re-maintenance message through the maintenance terminal.
[0027] The present disclosure also provides an image processing device, including:
[0028] The acquisition module is used to obtain the current maintenance image;
[0029] A duplicate checking module, used for inputting the current proxy image into a duplicate checking model to obtain the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point;
[0030] A matching module, used for inputting the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, and the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points into an image feature matching network, to obtain a matching result between the current proxy image and the target proxy image;
[0031] Among them, the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model.
[0032] The present disclosure also provides an electronic device, including:
[0033] processor;
[0034] a memory for storing instructions executable by the processor;
[0035] The processor is configured to execute the instructions to implement each step in the above method.
[0036] The embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the above method is implemented.
[0037] The present disclosure also provides a computer program product, including a computer program / instruction, which implements each step of the above method when executed by a processor.
[0038] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0039] In some embodiments of the present disclosure, a current proxy picture is obtained; the current proxy picture is input into a duplicate checking model to obtain a first interest point feature of the current proxy picture and a first surrounding image feature of each first interest point; the first interest point feature of the current proxy picture and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are input into an image feature matching network to obtain a matching result of the current proxy picture and the target proxy picture; wherein the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model; the present disclosure automatically extracts the interest point features of the proxy picture and the image features around the interest points through the duplicate checking model, and then inputs them into the image feature matching network to automatically perform image duplicate checking and matching, which improves the efficiency of image duplicate checking and the accuracy of image duplicate checking compared to the existing manual image duplicate checking method, which has high labor costs.
[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0042] Figure 1 A flowchart of an image processing method provided by an exemplary embodiment of the present disclosure;
[0043] Figure 2 A schematic diagram of a network structure of a duplicate checking model provided for an exemplary embodiment of the present disclosure;
[0044] Figure 3 A schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present disclosure;
[0045] Figure 4 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0046] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.
[0048] It should be noted that the user information involved in this disclosure includes but is not limited to: user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure comply with the relevant laws and regulations and do not violate public order and good morals.
[0049] The duplicate checking of maintenance pictures is mainly used in the auditing of maintenance work photos. It matches the newly uploaded maintenance work photos with the historical work photos to determine whether the maintenance personnel use the historical work photos to falsely return the receipt. If the duplicate checking of the newly added work photos fails, it means that the maintenance personnel may have processed the historical photos by cropping, filtering, rotating the pictures, etc., and uploaded them as proof of the new work order, but in fact did not perform the relevant work, which is a fraudulent behavior. The duplicate checking results of the pictures provide data support for the maintenance administrators of the municipal companies to audit the work of the maintenance personnel, so as to confirm whether the maintenance personnel's work is authentic and effective.
[0050] The existing technical solutions mainly rely on manual spot checks and audits, which randomly check the check-in photos of work orders in the maintenance management system every month to determine whether there are similar or identical photos. However, the monthly work order data of the maintenance system reaches 300,000 or more, with an average of 10 photos per work order, and a total of more than 3 million work order pictures per month. The manual review cycle is long and costly. At the same time, duplicate photos may appear in work orders in different regions and different sampling cycles. The above situations bring great difficulties to manual spot checks: as the time of manual duplicate checking continues to increase, administrators are prone to fatigue and misjudgment; multiple people spot checking at the same time cannot identify all the photos that have appeared, and manual spot checks have problems with low accuracy and completeness of duplicate checking.
[0051] The existing solution has the problem of long manual review cycle and high cost. The maintenance system generates more than 3 million photos every month, and manual spot checks are costly and inefficient.
[0052] The existing solution has the problem of incomplete duplicate checking samples. Due to the huge amount of existing work orders, manual spot checks can only be used for auditing, but they fail to cover all work orders and there are omissions. At the same time, manual spot checks are also limited by division of labor and cooperation. Multiple people spot checking at the same time cannot identify all photos that have appeared. For example, the same photo may be normal within the scope of auditor A's work order, but it is a duplicate photo within the scope of auditor B's work order. Therefore, the completeness of the duplicate photos manually checked is low, and the number of duplicate photos checked is far less than the actual situation.
[0053] The existing solution has the problem of low accuracy in duplicate checking. Maintenance personnel may process historical photos by cropping, filtering, rotating pictures, etc., and upload them as work photos for new work orders. However, similar photos may appear on work orders in different regions and different sampling cycles. Manual sampling is difficult and labor-intensive, and it is easy to make misjudgments. At the same time, multiple people sampling at the same time cannot identify all the photos that have appeared before, which ultimately leads to the problem of low accuracy in duplicate checking of maintenance pictures.
[0054] In response to the above technical problems, in some embodiments of the present disclosure, a current proxy picture is obtained; the current proxy picture is input into a duplicate checking model to obtain the first interest point features of the current proxy picture and the first surrounding image features of each first interest point; the first interest point features of the current proxy picture and the first surrounding image features of each first interest point, the second interest point features of the target proxy picture and the second surrounding image features of each second interest point are input into an image feature matching network to obtain a matching result of the current proxy picture and the target proxy picture; wherein the target proxy picture is any proxy picture in the image feature library, and the second interest point features of the target proxy picture and the second surrounding image features of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model; the present disclosure automatically extracts the interest point features of the proxy picture and the image features around the interest points through the duplicate checking model, and then inputs them into the image feature matching network to automatically perform image duplicate checking and matching. Compared with the existing manual method of image duplicate checking, the efficiency of image duplicate checking is improved, and the accuracy of image duplicate checking is improved, but the labor cost is high.
[0055] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.
[0056] Figure 1 FIG. 1 is a flow chart of an image processing method provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, the method includes:
[0057] S101: Get the current maintenance image;
[0058] S102: Input the current proxy image into a duplicate checking model to obtain a first point of interest feature of the current proxy image and a first surrounding image feature of each first point of interest;
[0059] S103: Inputting the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, and the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point into an image feature matching network to obtain a matching result between the current proxy image and the target proxy image;
[0060] Among them, the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model.
[0061] It should be noted that the execution subject of the above method can be a terminal device or a server.
[0062] Among them, the terminal device includes but is not limited to a mobile station (MS), a mobile terminal, a mobile phone, a handset and a portable equipment, etc. The terminal device can communicate with one or more core networks via a radio access network (RAN). For example, the terminal device can be a mobile phone (or a "cellular" phone), a computer with a wireless communication function, etc. The terminal device can also be a computer with a wireless transceiver function, a virtual reality (VR) terminal device, an AR terminal device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc., and the operating system installed on the terminal device includes but is not limited to: IOS, Android, windows, linux, Mac OS and other operating systems. In different networks, the terminal may be called different names, such as user equipment, mobile station, subscriber unit, station, cellular phone, personal digital assistant, wireless modem, wireless communication device, handheld device, laptop computer, cordless phone, wireless local loop station, television, etc. For the convenience of description, it is referred to as terminal equipment in this embodiment.
[0063] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly includes a processor, a hard disk, a memory, a system bus, etc., and a general computer architecture type.
[0064] In this embodiment, a current proxy picture is obtained; the current proxy picture is input into a duplicate checking model to obtain the first interest point features of the current proxy picture and the first surrounding image features of each first interest point; the first interest point features of the current proxy picture and the first surrounding image features of each first interest point, the second interest point features of the target proxy picture and the second surrounding image features of each second interest point are input into an image feature matching network to obtain a matching result of the current proxy picture and the target proxy picture; wherein the target proxy picture is any proxy picture in the image feature library, and the second interest point features of the target proxy picture and the second surrounding image features of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model; the present disclosure automatically extracts the interest point features of the proxy picture and the image features around the interest points through the duplicate checking model, and then inputs them into the image feature matching network to automatically perform image duplicate checking and matching. Compared with the existing manual method of image duplicate checking, the efficiency of image duplicate checking is improved, the accuracy of image duplicate checking is improved, and the labor cost is high.
[0065] The present invention uses a deep learning algorithm to extract image features from newly uploaded maintenance images, and compares them with the image feature library data of historical images, so as to realize duplicate checking of all maintenance images and improve the completeness rate of duplicate checking. Combined with the dynamic duplicate checking strategy, each region can make adjustments according to its own needs to make the duplicate checking results closer to actual needs and improve the accuracy of duplicate checking. At the same time, archived data is used as new samples to continuously train the duplicate checking model and continuously optimize the model parameters to achieve the purpose of improving model performance.
[0066] In some embodiments of the present disclosure, the current proxy image is input into the duplicate checking model to obtain the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point. Before using the duplicate checking model, it is necessary to build a duplicate checking model and an image feature library to quickly extract the proxy image features and pair the historical proxy images.
[0067] In the above embodiment, a duplicate checking model is constructed based on the SuperPoint image feature extraction deep learning algorithm and business scenarios, and the duplicate checking model is trained through a large number of maintenance production environment pictures, and the parameters are adjusted according to each training result, so that the trained model can quickly detect the interest point features of the newly added maintenance pictures and the high-dimensional vector descriptor of each interest point (i.e., the surrounding image features) for subsequent image pair matching. Among them, the descriptor is used to describe the image features around the point of interest. According to different business scenarios, the points of interest include but are not limited to the following: labels, helmets, resource equipment, engine equipment, etc. After the training of the duplicate checking model is completed, its weight parameter file can be obtained, which contains the feature information of different points of interest in different business scenarios.
[0068] The SuperPoint algorithm is a deep learning method for image feature extraction. It is mainly applicable to image registration, 3D reconstruction, object recognition and tracking, etc. It has the characteristics of fast running speed, strong robustness, and self-supervised learning. Based on the above characteristics, the duplicate detection model can quickly process different image scenes to complete image feature extraction; it can also extract image information more completely under various lighting, texture and occlusion conditions; at the same time, using self-supervised learning technology, the model is trained by comparing different geometric transformation versions of the same image, without the need for manual data annotation, reducing manual intervention.
[0069] The image feature library is constructed to store the feature information of points of interest after being detected and extracted by the duplicate checking model, so as to facilitate the subsequent matching of the feature information of the newly added maintenance pictures with the existing data and realize the intelligent duplicate checking of maintenance pictures. The image feature library is classified by business scenario. In actual use, it can be quickly screened according to the business scenario to improve the efficiency of duplicate checking of maintenance pictures.
[0070] Figure 2 A schematic diagram of a network structure of a duplicate checking model provided by an exemplary embodiment of the present disclosure. Figure 2 As shown, the duplicate checking model includes: an encoder, a feature point decoder and a descriptor decoder. Among them, the encoder is used to encode the input proxy image to obtain the image coding features. The feature point decoder is used to extract key points or feature points from the image, which have significant structures or information in the image. These feature points can be corner points, edge points, spots, etc., and finally output the interest point features of the proxy image. The descriptor decoder is used to generate a descriptor or feature vector for each interest point, which captures the image information of the area around the feature point, and finally outputs the surrounding image features of the interest point in the proxy image.
[0071] In this embodiment, after completing the above-mentioned duplicate checking model training and image feature library construction, the newly uploaded maintenance pictures are checked for duplicates. The duplicate checking process is based on a deep learning algorithm, and the matching relationship between image pairs is determined by the information of points of interest; finally, the optimal matching relationship is calculated iteratively to confirm whether the two pictures are similar.
[0072] The SuperGlue image matching algorithm is an image feature matching algorithm based on deep learning. The core idea is to imitate the process of human eyes matching two objects through self-attention and cross-attention. That is, when the eyes are looking for the similarity between two objects, they will constantly compare the feature points between the two objects and the feature points of the objects themselves, and regard feature matching as finding the corresponding similarity between two probability distributions.
[0073] In some disclosed embodiments, the image feature matching network includes: an attention graph neural network and an optimal matching layer. The first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point are input into the image feature matching network to obtain the matching result between the current proxy image and the target proxy image. One achievable method is to input the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point into the attention graph neural network inside the image feature matching network to obtain the first matching descriptor of the current proxy image and the second matching descriptor of the target proxy image; the first matching descriptor and the second matching descriptor are input into the optimal matching layer to obtain the matching result between the current proxy image and the target proxy image.
[0074] In the above embodiment, the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point are input into the attention graph neural network to obtain the first matching descriptor of the current proxy image and the second matching descriptor of the target proxy image. One achievable method is to input the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point into the feature point encoder respectively to obtain the first feature vector of the current proxy image and the second feature vector of the target proxy image; linearly project the first feature vector and the second feature vector respectively to obtain the first matching descriptor of the current proxy image and the second matching descriptor of the target proxy image. Among them, the attention graph neural network uses the feature point encoder to map the feature point position and the visual descriptor into a single vector, and then uses alternating self-attention and cross-attention layers to create a more powerful feature representation. The optimal matching layer creates an M×N score matrix, and through the Sinkhorn algorithm, iterates T times to find the optimal partial match, so that the matching relationship of the input image pair can be obtained.
[0075] Specifically, the desc0 descriptor of Figure A and the desc1 descriptor of Figure B can be obtained through feature point encoding, as shown in the following formula:
[0076]
[0077] Among them, i represents the sequence number of each key point; represents the initial state of each feature point i; d i represents the visual descriptor of each feature point; p iRepresents a three-dimensional data (x, y, c), x and y represent the position coordinates of the key point, and c represents the detection confidence; MLP enc (p i ) indicates that the key point position coordinates are mapped to the same dimension as the visual descriptor through a multi-layer perceptron. The key points and feature points mentioned above may be interest points, the visual descriptors may be surrounding image features, and the interest point features may be interest point position coordinates.
[0078] In the above embodiment, the first eigenvector and the second eigenvector are linearly projected to obtain the first matching descriptor of the current proxy image and the second matching descriptor of the target proxy image. The descriptors calculated by the graph neural network are linearly projected to obtain the matching descriptor f of the final image A. A And the matching descriptor f of Figure B B , as shown in the following formula:
[0079] f AorB =W· L x AorB +b
[0080] Among them, L represents the number of repetitions; W represents the weight matrix; and b represents the offset.
[0081] In some embodiments of the present disclosure, the first matching descriptor and the second matching descriptor are input into the optimal matching layer to obtain the matching result between the current proxy image and the target proxy image. One achievable method is to perform an inner product between the first matching descriptor and the second matching descriptor to obtain an allocation matrix, wherein the allocation matrix is used to represent the similarity between the first matching descriptor and the second matching descriptor; count the number of element values of the allocation matrix that are greater than the confidence threshold; and determine the matching result between the current proxy image and the target proxy image based on the number of element values of the allocation matrix that are greater than the confidence threshold. Among them, the matching descriptor f of the two images obtained by the attention graph neural network is A and f B , the best matching layer pair f A and f B Do the inner product to calculate a distribution matrix S∈R M×N , this assignment matrix is used to represent the similarity of matching descriptors by maximizing the total score ∑ i,j S i,j P i,j , the optimal transmission problem is solved to calculate the allocation matrix P. Based on the Sinkhorn algorithm, entropy regularization is introduced to complete the efficient iterative calculation of the optimal transmission problem between two distributions, that is, to calculate the allocation matrix result.
[0082] It should be noted that the similarity of the image pairs can be obtained based on the distribution matrix results. The non-zero elements in the distribution matrix (or elements greater than a certain threshold) indicate that there is a matching relationship between the feature points of the corresponding rows and columns: if the element value of the i-th row and j-th column of the distribution matrix is high, it means that the i-th feature point in image A and the j-th feature point in image B are likely to match; and the number of non-zero elements in the distribution matrix directly reflects the total number of matching point pairs. By setting the confidence threshold of the distribution matrix, high-confidence matching point pairs can be screened out from the results to avoid other influences and make the results more reliable.
[0083] In some embodiments of the present disclosure, the matching result between the current proxy picture and the target proxy picture is determined based on the number of element values of the allocation matrix that are greater than the confidence threshold. One achievable method is to determine that the current proxy picture and the target proxy picture are successfully matched when the number is greater than or equal to the number threshold; and to determine that the current proxy picture and the target proxy picture are not successfully matched when the number is less than the number threshold. It should be noted that the present disclosure does not limit the confidence threshold and the number threshold, and the confidence threshold and the number threshold can be adjusted according to actual conditions.
[0084] In some embodiments of the present disclosure, when there is a maintenance picture in the image feature library that matches the current maintenance picture successfully, a re-maintenance message is generated; the re-maintenance message is sent to the maintenance terminal so that the maintenance user can view the re-maintenance message through the maintenance terminal. Among them, the maintenance picture duplicate checking method disclosed in the present disclosure can realize automated intelligent duplicate checking, and thereby replace the existing manual spot check method, thereby reducing labor costs. When the user submits a work order, the duplicate checking model is used to quickly extract image features from the newly uploaded maintenance picture in combination with the business scenario of the current work order, and the image features are matched with the historical pictures in the image feature library, and the image duplicate checking results are output according to the matching results. Through the duplicate checking results, early warnings can be given to maintenance personnel in a timely manner, and maintenance personnel can be continuously guided to standardize their work and improve the quality of maintenance work.
[0085] It should be noted that manual sampling can be used to check whether the accuracy of the duplicate checking results meets the expected effect, and the algorithm can be optimized and adjusted based on the verification results. By continuously iterating the algorithm design and dynamically adjusting the parameters in various business scenarios, the duplicate checking results can be more in line with production needs, providing strong data support for the quality of maintenance operations.
[0086] In some embodiments of the present disclosure, newly added archived maintenance images have a positive feedback effect on the training of the duplicate checking model, and the newly added archived data is supplemented into the duplicate checking model data set as an incremental sample. The incremental sample data is used to train the duplicate checking model, so as to achieve the purpose of dynamically adjusting and optimizing the duplicate checking model, so that the image feature extraction results are more and more accurate and reliable. At the same time, the extracted image feature data is entered into the image feature library, and the image feature library data is continuously expanded to ensure the integrity of the work order duplicate checking. Through the continuous archived data feedback strategy, administrators in various cities can adjust the duplicate checking strategy according to the latest duplicate checking situation, so as to achieve the purpose of improving both the efficiency and accuracy of duplicate checking.
[0087] In the above-mentioned method embodiment of the present disclosure, a current proxy picture is obtained; the current proxy picture is input into a duplicate checking model to obtain the first interest point feature of the current proxy picture and the first surrounding image feature of each first interest point; the first interest point feature of the current proxy picture and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are input into an image feature matching network to obtain a matching result of the current proxy picture and the target proxy picture; wherein the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model; the present disclosure automatically extracts the interest point features and the image features around the interest points of the proxy picture through the duplicate checking model, and then inputs them into the image feature matching network to automatically perform image duplicate checking and matching, which improves the efficiency of image duplicate checking and the accuracy of image duplicate checking compared to the existing manual method of image duplicate checking, but with high labor costs.
[0088] Figure 3 FIG. 3 is a schematic diagram of the structure of an image processing device 30 provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the image processing device 30 includes: an acquisition module 31, a duplicate checking module 32 and a matching module 33.
[0089] The acquisition module 31 is used to acquire the current maintenance image;
[0090] A duplicate checking module 32 is used to input the current proxy image into a duplicate checking model to obtain the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point;
[0091] The matching module 33 is used to input the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, and the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point into the image feature matching network to obtain the matching result of the current proxy image and the target proxy image;
[0092] Among them, the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model.
[0093] Optionally, the image feature matching network includes: an attention graph neural network and an optimal matching layer. When the matching module 33 inputs the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point into the image feature matching network, and obtains the matching result of the current proxy image and the target proxy image, it is used to:
[0094] Inside the image feature matching network, the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point are input into the attention graph neural network to obtain the first matching descriptor of the current proxy image and the second matching descriptor of the target proxy image;
[0095] The first matching descriptor and the second matching descriptor are input into the optimal matching layer to obtain the matching result between the current proxy image and the target proxy image.
[0096] Optionally, when the matching module 33 inputs the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point into the attention graph neural network to obtain the first matching descriptor of the current proxy image and the second matching descriptor of the target proxy image, it is used to:
[0097] Input the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, and the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point into the feature point encoder respectively, to obtain the first feature vector of the current proxy image and the second feature vector of the target proxy image;
[0098] The first eigenvector and the second eigenvector are linearly projected respectively to obtain a first matching descriptor of the current proxy image and a second matching descriptor of the target proxy image.
[0099] Optionally, when the matching module 33 inputs the first matching descriptor and the second matching descriptor into the optimal matching layer to obtain the matching result between the current proxy image and the target proxy image, it is used to:
[0100] Taking the inner product of the first matching descriptor and the second matching descriptor to obtain a distribution matrix, wherein the distribution matrix is used to represent the similarity between the first matching descriptor and the second matching descriptor;
[0101] Count the number of elements of the distribution matrix that are greater than the confidence threshold;
[0102] According to the number of element values of the allocation matrix that are greater than the confidence threshold, the matching result between the current proxy image and the target proxy image is determined.
[0103] Optionally, when determining the matching result between the current proxy image and the target proxy image according to the number of element values of the allocation matrix that are greater than the confidence threshold, the matching module 33 is configured to:
[0104] If the number is greater than or equal to the number threshold, it is determined that the current proxy image matches the target proxy image successfully;
[0105] When the number is less than the number threshold, it is determined that the current proxy image and the target proxy image are not successfully matched.
[0106] Optionally, the matching module 33 may also be used to:
[0107] If there is a proxy maintenance picture in the image feature library that matches the current proxy maintenance picture successfully, a re-maintenance message is generated;
[0108] The re-maintenance message is sent to the maintenance terminal so that the maintenance user can view the re-maintenance message through the maintenance terminal.
[0109] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0110] Figure 4 FIG. 1 is a schematic diagram of a structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the electronic device includes: a memory 41 and a processor 42. In addition, the electronic device also includes a power supply component 43 and a communication component 44.
[0111] The memory 41 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.
[0112] The memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0113] The communication component 44 is used for data transmission with other devices.
[0114] The processor 42 can execute the computer instructions stored in the memory 41 to:
[0115] Get the current maintenance image;
[0116] Input the current proxy image into the duplicate checking model to obtain the first interest point feature of the current proxy image and the first surrounding image features of each first interest point;
[0117] Input the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point, and the second interest point feature of the target proxy image and the second surrounding image feature of each second interest point into the image feature matching network to obtain the matching result of the current proxy image and the target proxy image;
[0118] Among them, the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model.
[0119] Accordingly, the present disclosure also provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 1 Each step in the method embodiment.
[0120] Accordingly, the present disclosure also provides a computer program product, the computer program product includes a computer program / instruction, the computer program / instruction is executed by a processor Figure 1 Each step in the method embodiment.
[0121] Above Figure 4The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0122] Above Figure 4 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.
[0123] The electronic device also includes a display screen and an audio component.
[0124] The display screen includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
[0125] The audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.
[0126] In the above-mentioned device, equipment, storage medium and computer program product embodiments of the present disclosure, a current proxy picture is obtained; the current proxy picture is input into a duplicate checking model to obtain the first interest point feature of the current proxy picture and the first surrounding image feature of each first interest point; the first interest point feature of the current proxy picture and the first surrounding image feature of each first interest point, the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are input into an image feature matching network to obtain a matching result of the current proxy picture and the target proxy picture; wherein the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model; the present disclosure automatically extracts the interest point features of the proxy picture and the image features around the interest points through the duplicate checking model, and then inputs them into the image feature matching network to automatically perform image duplicate checking and matching, which improves the efficiency of image duplicate checking and the accuracy of image duplicate checking compared to the existing manual method of image duplicate checking, and the labor cost is relatively high.
[0127] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0128] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.
[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0131] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0132] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0134] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0135] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image processing method, characterized in that: include: Get the current maintenance image; Input the current proxy image into a duplicate checking model to obtain a first point of interest feature of the current proxy image and a first surrounding image feature of each first point of interest; Input the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, and the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points into an image feature matching network to obtain a matching result between the current proxy image and the target proxy image; Among them, the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model.
2. The method according to claim 1, characterized in that The image feature matching network includes: an attention graph neural network and an optimal matching layer, wherein the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points are input into the image feature matching network to obtain a matching result between the current proxy image and the target proxy image, including: Inside the image feature matching network, the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points are input into the attention graph neural network to obtain a first matching descriptor of the current proxy image and a second matching descriptor of the target proxy image; The first matching descriptor and the second matching descriptor are input into the optimal matching layer to obtain a matching result between the current proxy image and the target proxy image.
3. The method according to claim 2, characterized in that The step of inputting the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, and the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points into the attention graph neural network to obtain a first matching descriptor of the current proxy image and a second matching descriptor of the target proxy image, comprises: Input the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, and the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points into a feature point encoder respectively, to obtain a first feature vector of the current proxy image and a second feature vector of the target proxy image; The first feature vector and the second feature vector are linearly projected respectively to obtain the first matching descriptor of the current proxy image and the second matching descriptor of the target proxy image.
4. The method according to claim 2, characterized in that: The step of inputting the first matching descriptor and the second matching descriptor into the optimal matching layer to obtain a matching result between the current proxy image and the target proxy image includes: Taking the first matching descriptor and the second matching descriptor as an inner product to obtain a distribution matrix, wherein the distribution matrix is used to represent the similarity between the first matching descriptor and the second matching descriptor; Counting the number of element values of the allocation matrix that are greater than a confidence threshold; The matching result between the current proxy image and the target proxy image is determined according to the number of element values of the allocation matrix that are greater than the confidence threshold.
5. The method according to claim 4, characterized in that The determining, according to the number of element values of the allocation matrix that are greater than a confidence threshold, a matching result between the current proxy image and the target proxy image, comprises: If the number is greater than or equal to the number threshold, it is determined that the current proxy image matches the target proxy image successfully; When the number is less than the number threshold, it is determined that the current proxy image and the target proxy image are not successfully matched.
6. The method according to claim 1, characterized in that The method further comprises: If there is a proxy image in the image feature library that matches the current proxy image successfully, a re-maintenance message is generated; The re-maintenance message is sent to the maintenance terminal so that the maintenance user can view the re-maintenance message through the maintenance terminal.
7. An image processing device, characterized in that: include: The acquisition module is used to obtain the current maintenance image; A duplicate checking module, used for inputting the current proxy image into a duplicate checking model to obtain the first interest point feature of the current proxy image and the first surrounding image feature of each first interest point; A matching module, used for inputting the first interest point feature of the current proxy image and the first surrounding image feature of each of the first interest points, and the second interest point feature of the target proxy image and the second surrounding image feature of each of the second interest points into an image feature matching network, to obtain a matching result between the current proxy image and the target proxy image; Among them, the target proxy picture is any proxy picture in the image feature library, and the second interest point feature of the target proxy picture and the second surrounding image feature of each second interest point are features obtained by inputting the target proxy picture into the duplicate checking model.
8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement each step in the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.