A locking device detection system and method
By embedding a multi-label classification layer and a transformation inference layer into the locking device detection system, the problems of accuracy and universality in locking device detection are solved, and efficient locking device detection is achieved.
Patent Information
- Application Number
- CN202411045874.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Existing detection schemes for locking devices suffer from problems such as decreased positioning accuracy, the need for additional training costs, and equipment specificity.
A locking device detection system is adopted, including an embedded multi-label classification layer, a transformation unit, an inference unit, a filtering unit, and a classification unit. By embedding the multi-label classification layer into the fusion layer of the target detection model, the multi-label classification layer and the first K layers of the target detection model are transformed into the first inference layer, and the last NK layers are transformed into the second inference layer. Image features and locking device features are calculated and filtered to obtain the confidence level and classification result of the locking device.
It accelerates inference for locking device detection, improves localization accuracy without requiring additional training costs, and enhances the versatility of detection.
Smart Images

Figure CN118570203B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent detection technology, and in particular to a locking device detection system and method thereof. Background Art
[0002] In the field of aerospace, locking devices are used to fix and lock various components, equipment or tools in spacecraft or aircraft. There are many types of locking devices, and the functions of each locking device may be different. The differences between multiple locking devices are very small. Therefore, it is necessary to quickly infer the newly produced locking devices and accurately locate each locking device. In the prior art, the model distillation solution is used to accelerate reasoning, but the positioning accuracy decreases and requires additional training costs; the model quantization solution is used to accelerate reasoning, but the versatility is poor; the model pruning solution is used to accelerate reasoning, which requires specific equipment. Summary of the invention
[0003] The present invention provides a locking device detection system and method thereof, which are used to solve the defects in the prior art that the solution of model distillation is used to accelerate reasoning, the positioning accuracy is reduced, and additional training costs are required; the solution of model quantization is used to accelerate reasoning, the versatility is poor; and the solution of model pruning is used to accelerate reasoning, which requires specific equipment.
[0004] The present invention provides a locking device detection system, comprising a locking device detection center, an embedding unit, a conversion unit, an inference unit, a filtering unit and a classification unit; the locking device detection center is respectively connected with the embedding unit, the conversion unit, the inference unit, the filtering unit and the classification unit to store and manage the data of each unit;
[0005] The embedding unit is used to embed a multi-label classification layer in any one layer of the fusion layer of the target detection model, the target detection model includes N layers, N is a natural number greater than or equal to 1, and the target detection model is trained based on the sample image and multiple sample category locking devices, and the classification result labels of the sample image;
[0006] The conversion unit is used to convert the multi-label classification layer and the first K layers before the multi-label classification layer in the target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the target detection model into a second inference layer, where K is a natural number greater than or equal to 1;
[0007] The inference unit is used to input the target image and the preset M target category locking devices into the first inference layer, and obtain the image features and M locking device features corresponding to the target image output by the first inference layer, as well as the confidence of the M target category locking devices, where M is a natural number greater than or equal to 1;
[0008] The filtering unit is used to filter the M locking device features based on the confidence of the M target category locking devices to obtain filtered m locking device features, where m is a non-negative integer;
[0009] The classification unit is used to input the filtered m locking device features and the image features into the second reasoning layer when the value of m is not zero, so as to obtain the classification result of the target image output by the second reasoning layer.
[0010] The present invention also provides a locking device detection method, comprising:
[0011] Embed a multi-label classification layer in any one layer of the fusion layer of the target detection model, wherein the target detection model includes N layers, N is a natural number greater than or equal to 1, and the target detection model is trained based on the sample image and multiple sample category locking devices, and the classification result labels of the sample image;
[0012] Convert the multi-label classification layer and the first K layers before the multi-label classification layer in the target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the target detection model into a second inference layer, where K is a natural number greater than or equal to 1;
[0013] Inputting a target image and M preset target category locking devices into the first reasoning layer, obtaining image features and M locking device features corresponding to the target image output by the first reasoning layer, and confidences of the M target category locking devices, where M is a natural number greater than or equal to 1;
[0014] Based on the confidences of the M target category locking devices, the M locking device features are filtered to obtain m filtered locking device features, where m is a non-negative integer;
[0015] When the value of m is not zero, the filtered m locking device features and the image features are input into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer.
[0016] According to the locking device detection method provided by the present invention, the target image and the preset M target category locking devices are input into the first reasoning layer, and the image features and M locking device features corresponding to the target image output by the first reasoning layer, as well as the confidence of the M target category locking devices, are obtained, including:
[0017] Input the target image and the preset M target category locking devices into the first K layers, and obtain the image features and M locking device features corresponding to the target image output by the first K layers;
[0018] Based on the multi-label classification layer, the similarity between each locking device feature and the image feature of the corresponding area is calculated, and based on the similarity between each locking device feature and the image feature of the corresponding area, the category confidence of each locking device feature is calculated.
[0019] According to the locking device detection method provided by the present invention, the M locking device features are filtered based on the confidence of the M target category locking devices to obtain the filtered m locking device features, including:
[0020] Based on the confidences of the M target category locking devices and a preset threshold, the M locking device features are filtered to obtain the filtered m locking device features.
[0021] According to the locking device detection method provided by the present invention, after obtaining the filtered m locking device features, the method further includes:
[0022] When the value of m is zero, the target detection model directly outputs the classification result of the target image as empty.
[0023] According to the locking device detection method provided by the present invention, the training process of the target detection model includes:
[0024] Acquire a sample image and a plurality of sample category locking devices, and determine a classification result label of the sample image;
[0025] Using the sample image and multiple sample category locking devices as training samples, and using the classification result labels of the sample image as sample labels to train an initial target detection model;
[0026] The parameters of the initial target detection model are iteratively optimized to obtain the target detection model.
[0027] According to the locking device detection method provided by the present invention, the initial target detection model includes a first initial reasoning layer and a second initial reasoning layer, the first initial reasoning layer includes an initial multi-label classification layer and an initial front K layer, the initial front K layer refers to the K layers in the initial target detection model located before the initial multi-label classification layer, and the initial target detection model includes N layers;
[0028] Correspondingly, the training of the initial target detection model includes:
[0029] Inputting a sample image and a plurality of sample category locking devices into the initial front K layers, and obtaining a sample image feature and a plurality of sample category locking device features corresponding to the sample image output by the initial front K layers;
[0030] Based on the initial multi-label classification layer, calculating the similarity between each sample locking device feature and the sample image feature of the corresponding area;
[0031] Calculating a loss function value based on the similarity between each locking device feature and the image feature of the corresponding area and the classification result label of the sample image;
[0032] Based on the loss function value, the initial multi-label classification layer is trained.
[0033] The present invention also provides an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby implementing any of the above-mentioned locking device detection methods.
[0034] The present invention also provides a non-transitory computer-readable storage medium, wherein a computer software program is stored in the storage medium, and when the computer software program is executed by a processor, the locking device detection method described in any one of the above is implemented.
[0035] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the locking device detection method as described above is implemented.
[0036] According to the locking device detection system of the embodiment of the present invention, a multi-label classification layer is embedded in any layer of the fusion layer of the target detection model, the multi-label classification layer and the first K layers in the target detection model are converted into the first reasoning layer, and the last NK layers in the target detection model are converted into the second reasoning layer. The target image and preset M target category locking devices are input into the first reasoning layer to obtain image features and M locking device features corresponding to the target image, as well as the confidences of the M target category locking devices. Based on the confidences of the M target category locking devices, the M locking device features are filtered to obtain filtered m locking device features. When the m value is not zero, the filtered m locking device features and image features are input into the second reasoning layer to obtain the classification result of the target image. The reasoning acceleration effect is good, no additional training cost is required, the positioning accuracy is improved, and the versatility is strong. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a structural schematic diagram of the locking device detection system provided by the present invention;
[0038] Figure 2 It is a schematic flow chart of the locking device detection method provided by the present invention;
[0039] Figure 3 It is a schematic diagram of the process of deploying the target detection model provided by the present invention;
[0040] Figure 4 It is a schematic diagram of a flow chart of filtering multiple locking device features based on the confidence of multiple target category locking devices provided by the present invention;
[0041] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0043] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0044] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.
[0045] Reference Figure 1 , Figure 1 It is a structural schematic diagram of the locking device detection system provided by the present invention, and the locking device detection system includes a locking device detection middle station, an embedding unit, a conversion unit, an inference unit, a filtering unit and a classification unit; the locking device detection middle station is respectively connected to the embedding unit, the conversion unit, the inference unit, the filtering unit and the classification unit to store and manage the data of each unit.
[0046] Optionally, the embedding unit embeds a multi-label classification layer in any layer of the fusion layer of the target detection model, wherein the target detection model includes N layers, N is a natural number greater than or equal to 1, and the target detection model is trained based on sample images and multiple sample category locking devices, as well as classification result labels of the sample images.
[0047] Optionally, the conversion unit converts the multi-label classification layer and the first K layers before the multi-label classification layer in the target detection model into a first inference layer, and converts the NK layer after the multi-label classification layer in the target detection model into a second inference layer, where K is a natural number greater than or equal to 1.
[0048] Optionally, the reasoning unit inputs the target image and M preset target category locking devices into the first reasoning layer, obtains image features and M locking device features corresponding to the target image output by the first reasoning layer, and confidence levels of the M target category locking devices, where M is a natural number greater than or equal to 1.
[0049] Optionally, the filtering unit filters the M locking device features based on the confidences of the M target category locking devices to obtain m filtered locking device features, where m is a non-negative integer.
[0050] Optionally, the classification unit is used to input the filtered m locking device features and image features into the second reasoning layer when the value of m is not zero, so as to obtain a classification result of the target image output by the second reasoning layer.
[0051] According to the locking device detection system of the embodiment of the present invention, a multi-label classification layer is embedded in any layer of the fusion layer of the target detection model, the multi-label classification layer and the first K layers in the target detection model are converted into the first reasoning layer, and the last NK layers in the target detection model are converted into the second reasoning layer. The target image and preset M target category locking devices are input into the first reasoning layer to obtain image features and M locking device features corresponding to the target image, as well as the confidences of the M target category locking devices. Based on the confidences of the M target category locking devices, the M locking device features are filtered to obtain filtered m locking device features. When the m value is not zero, the filtered m locking device features and image features are input into the second reasoning layer to obtain the classification result of the target image. The reasoning acceleration effect is good, no additional training cost is required, the positioning accuracy is improved, and the versatility is strong.
[0052] Figure 2 Schematic diagram of the flow of the locking device detection method provided by the embodiment of the present invention. Figure 2 As shown, a locking device detection method is provided, comprising the following steps: Steps 110 to 150. The method flow steps are only used as a possible implementation of the present invention.
[0053] Step 110: embed a multi-label classification layer in any layer of the fusion layer of the target detection model, where the target detection model includes N layers, where N is a natural number greater than or equal to 1.
[0054] Among them, the object detection model includes an image feature encoder, a fusion layer and a decoder.
[0055] It should be noted that the image features in the decoder remain unchanged. Embedding the multi-label classification layer in the decoder is equivalent to embedding it at the end of the fusion layer, which has less acceleration and is not a recommended embedding position in the application. The image feature encoder is used to encode image features and is implicitly aligned with the locking device features. Embedding the multi-label classification layer in the image feature encoder will result in poor similarity between the image features input in the multi-label classification layer and the locking device features, and reduced classification reliability. Therefore, in the target detection model, the embedding position can be selected in any layer of the fusion layer.
[0056] Preferably, a multi-label classification layer is embedded in the first layer of the fusion layer, which has a good inference acceleration effect and strong classification reliability.
[0057] Step 120: convert the multi-label classification layer and the first K layers before the multi-label classification layer in the target detection model into the first inference layer, and convert the NK layer after the multi-label classification layer in the target detection model into the second inference layer, where K is a natural number greater than or equal to 1.
[0058] Figure 3The following is a flow chart of the target detection model deployment process provided by the embodiment of the present invention. Figure 3 As shown, the multi-label classification layer and the first K layers of the target detection model are converted to the first inference layer, corresponding to Figure 3 Engine1 in the target detection model converts the NK layer after the target detection model into the second reasoning layer, corresponding to Figure 3 MediumEngine2.
[0059] Step 130: Input the target image and the preset M target category locking devices into the first reasoning layer, obtain the image features and M locking device features corresponding to the target image output by the first reasoning layer, and the confidence of the M target category locking devices, where M is a natural number greater than or equal to 1.
[0060] Optionally, the original M target category locking devices in the target detection model are input into the locking device encoder to obtain M locking device features.
[0061] Optionally, the image features corresponding to the target image are region-level image features, including image features of each region of the target image.
[0062] In some embodiments, the target image and the preset M target category locking devices are input into the first reasoning layer, and the image features and the M locking device features corresponding to the target image output by the first reasoning layer, as well as the confidence of the M target category locking devices, are obtained, including:
[0063] Input the target image and the preset M target category locking devices into the first K layers, and obtain the image features corresponding to the target image output by the first K layers and the M locking device features;
[0064] Based on the multi-label classification layer, the similarity between each locking device feature and the image feature of the corresponding area is calculated, and based on the similarity between each locking device feature and the image feature of the corresponding area, the category confidence of each locking device feature is calculated.
[0065] Optionally, the preset M target category locking devices are determined according to user input. After the target image passes through the first K layers, an image feature with a dimension of H×W×C is obtained, where H and W represent the height and width of the image, and C represents the number of features of the image. The spatial dimensions of the image feature are flattened and transposed to obtain the regional level image feature (HW×C); the preset M target category locking devices are processed using a locking device encoder to obtain a locking device feature (M×C); the regional image features and the locking device features are matrix multiplied to obtain a similarity matrix (M×HW); an aggregation function is used to aggregate the regional level single label classification results into the image level multi-label classification results to obtain the confidence of each target category locking device.
[0066] It should be noted that the similarity matrix contains fine-grained regional image classification information. In order to aggregate this fine-grained information, the embodiment of the present invention adopts the maximum pooling operation along the HW dimension to obtain global multi-label classification information of size M×1. Due to the use of the maximum pooling operation, as long as there is an area in the target image that is similar to the locking device label, the corresponding locking device category label can be retained during the reasoning process, thereby avoiding missed detection and ensuring that the detection accuracy of reasoning and training is consistent.
[0067] Step 140: Based on the confidences of the M target category locking devices, the M locking device features are filtered to obtain m filtered locking device features, where m is a non-negative integer.
[0068] It can be understood that by filtering the M locking device features, the amount of data processing can be reduced, thereby improving the detection efficiency.
[0069] Optionally, the value range of m is [0,M].
[0070] Figure 4 A schematic diagram of a process for filtering multiple locking device features based on the confidence of multiple target category locking devices provided by an embodiment of the present invention. Figure 4 As shown, in some embodiments, based on the confidence of the M target category locking devices, the M locking device features are filtered to obtain the filtered m locking device features, including:
[0071] Based on the confidences of the locking devices of the M target categories and a preset threshold, the M locking device features are filtered to obtain m filtered locking device features.
[0072] Optionally, a category filtering mechanism is used to filter the M locking device features.
[0073] For example, M is 5, and the confidence levels of the M target category locking devices are 0.8, 0.1, 0.7, 0.1, and 0.8, respectively. Combined with a preset threshold (such as 0.6), a bool mask is generated. The mask indicates whether the corresponding locking device category exists in the target image. The mask is used to filter the 5 locking device features corresponding to the 5 target category locking devices to obtain 3 locking device features.
[0074] Step 150: when the value of m is not zero, the filtered m locking device features and image features are input to the second reasoning layer to obtain a classification result of the target image output by the second reasoning layer;
[0075] The target detection model is trained based on sample images and multiple sample category locking devices, as well as classification result labels of the sample images.
[0076] It can be understood that by inputting the filtered m locking device features and image features into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer, the computational overhead of the second reasoning layer can be reduced and reasoning acceleration can be achieved.
[0077] In some embodiments, after obtaining the filtered m locking device features, the method further includes:
[0078] When the value of m is zero, the target detection model directly outputs the classification result of the target image as empty. It can be understood that when the value of m is zero, it indicates that there is no corresponding object in any locking device in the target image, and the processing of the second inference layer can be bypassed and directly returned, and the result is empty; in this case, the single calculation cost of reasoning is reduced from the first inference layer + the second inference layer to the first inference layer, achieving inference acceleration.
[0079] Optionally, the locking device detection method includes two parts: embedded multi-label classification layer training and inference acceleration; a multi-label classification layer is inserted during the pre-training process of the target detection model, thereby avoiding the additional training overhead of the previous acceleration scheme; in the multi-label classification loss layer, the similarity between the image region features and the target category locking device features is used as the classification score, so that the multi-label classification layer inserted after pre-training has fine-grained classification capabilities at the regional level; during the inference process, when the model runs to the multi-label classification layer, the similarity between the image region features and the locking device features of the target category is calculated, and the multi-label confidence is obtained through global aggregation. According to the set threshold, the target categories below the threshold are deleted in the subsequent inference process, thereby achieving inference acceleration without affecting the accuracy of the detection results.
[0080] In an embodiment of the present invention, a multi-label classification layer is embedded in any layer of the fusion layer of the target detection model, the multi-label classification layer and the first K layers in the target detection model are converted into a first reasoning layer, and the last NK layers in the target detection model are converted into a second reasoning layer. The target image and the preset M target category locking devices are input into the first reasoning layer to obtain image features and M locking device features corresponding to the target image, as well as the confidences of the M target category locking devices. Based on the confidences of the M target category locking devices, the M locking device features are filtered to obtain the filtered m locking device features. When the m value is not zero, the filtered m locking device features and the image features are input into the second reasoning layer to obtain the classification result of the target image. The reasoning acceleration effect is good, no additional training cost is required, the positioning accuracy is improved, and the versatility is strong.
[0081] In some embodiments, the training process of the object detection model includes:
[0082] Acquire a sample image and a plurality of sample category locking devices, and determine a classification result label of the sample image;
[0083] Using sample images and multiple sample category locking devices as training samples, and using classification result labels of sample images as sample labels to train an initial target detection model;
[0084] The parameters of the initial target detection model are iteratively optimized to obtain the target detection model.
[0085] Optionally, the initial object detection model includes an initial image feature encoder, an initial fusion layer, and an initial decoder.
[0086] In some embodiments, the initial target detection model includes a first initial reasoning layer and a second initial reasoning layer, the first initial reasoning layer includes an initial multi-label classification layer and an initial first K layer, the initial first K layer refers to the K layers before the initial multi-label classification layer in the initial target detection model, and the initial target detection model includes N layers;
[0087] Correspondingly, train the initial target detection model, including:
[0088] Inputting the sample image and the plurality of sample category locking devices into the initial first K layers, obtaining the sample image features and the plurality of sample category locking device features corresponding to the sample image output by the initial first K layers;
[0089] Based on the initial multi-label classification layer, calculating the similarity between each sample locking device feature and the sample image feature of the corresponding area;
[0090] Calculating a loss function value based on the similarity between each locking device feature and the image feature of the corresponding area and the classification result label of the sample image;
[0091] Based on the loss function value, train the initial multi-label classification layer.
[0092] It should be noted that the initial multi-label classification layer is a parameter-free layer and no additional training parameters are introduced.
[0093] Optionally, during the process of training the initial target detection model, a multi-label classification loss function is used to calculate the loss function value.
[0094] Among them, the calculation formula of the multi-label classification loss function is as follows:
[0095] ;
[0096] Wherein, N represents the number of sample category lock devices, p(yi) represents the probability of the existence of sample category lock device i after maximum pooling aggregation, yi represents the true value of sample category lock device i, yi is 0 or 1, 0 indicates that the sample category lock device i does not exist in the sample image, and 1 indicates that the sample category lock device i exists in the sample image.
[0097] It can be understood that by introducing a multi-label loss function in the training process of the object detection model, the training of the multi-label classification layer can be completed simultaneously without additional training cost.
[0098] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the locking device detection method, which includes:
[0099] Embed a multi-label classification layer in any one layer of the fusion layer of the target detection model, wherein the target detection model includes N layers, N is a natural number greater than or equal to 1, and the target detection model is trained based on the sample image and multiple sample category locking devices, and the classification result labels of the sample image;
[0100] Convert the multi-label classification layer and the first K layers before the multi-label classification layer in the target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the target detection model into a second inference layer, where K is a natural number greater than or equal to 1;
[0101] Inputting a target image and M preset target category locking devices into the first reasoning layer, obtaining image features and M locking device features corresponding to the target image output by the first reasoning layer, and confidences of the M target category locking devices, where M is a natural number greater than or equal to 1;
[0102] Based on the confidences of the M target category locking devices, the M locking device features are filtered to obtain m filtered locking device features, where m is a non-negative integer;
[0103] When the value of m is not zero, the filtered m locking device features and the image features are input into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer.
[0104] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0105] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the locking device detection method provided by the above methods, the method comprising:
[0106] Embed a multi-label classification layer in any one layer of the fusion layer of the target detection model, wherein the target detection model includes N layers, N is a natural number greater than or equal to 1, and the target detection model is trained based on the sample image and multiple sample category locking devices, and the classification result labels of the sample image;
[0107] Convert the multi-label classification layer and the first K layers before the multi-label classification layer in the target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the target detection model into a second inference layer, where K is a natural number greater than or equal to 1;
[0108] Inputting a target image and M preset target category locking devices into the first reasoning layer, obtaining image features and M locking device features corresponding to the target image output by the first reasoning layer, and confidences of the M target category locking devices, where M is a natural number greater than or equal to 1;
[0109] Based on the confidences of the M target category locking devices, the M locking device features are filtered to obtain m filtered locking device features, where m is a non-negative integer;
[0110] When the value of m is not zero, the filtered m locking device features and the image features are input into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer.
[0111] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned locking device detection method, the method comprising:
[0112] Embed a multi-label classification layer in any one layer of the fusion layer of the target detection model, wherein the target detection model includes N layers, N is a natural number greater than or equal to 1, and the target detection model is trained based on the sample image and multiple sample category locking devices, and the classification result labels of the sample image;
[0113] Convert the multi-label classification layer and the first K layers before the multi-label classification layer in the target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the target detection model into a second inference layer, where K is a natural number greater than or equal to 1;
[0114] Inputting a target image and M preset target category locking devices into the first reasoning layer, obtaining image features and M locking device features corresponding to the target image output by the first reasoning layer, and confidences of the M target category locking devices, where M is a natural number greater than or equal to 1;
[0115] Based on the confidences of the M target category locking devices, the M locking device features are filtered to obtain m filtered locking device features, where m is a non-negative integer;
[0116] When the value of m is not zero, the filtered m locking device features and the image features are input into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer.
[0117] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.
[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A locking device detection system, characterized in that: The locking device detection system includes a locking device detection center, an embedding unit, a conversion unit, an inference unit, a filtering unit, and a classification unit; the locking device detection center is connected to the embedding unit, the conversion unit, the inference unit, the filtering unit, and the classification unit respectively to store and manage data of each unit; The embedding unit is configured to embed a multi-label classification layer in any one of the fusion layers of the target detection model, wherein the target detection model includes N layers, where N is a natural number greater than or equal to 1, and the target detection model is trained based on a sample image and a plurality of sample category locking devices, as well as classification result labels of the sample image; The conversion unit is configured to convert the multi-label classification layer and the first K layers in the target detection model located before the multi-label classification layer into a first inference layer, and convert the NK layers in the target detection model located after the multi-label classification layer into a second inference layer, where K is a natural number greater than or equal to 1; The inference unit is configured to input a target image and M preset target category locking devices into the first inference layer, and obtain image features corresponding to the target image and M locking device features output by the first inference layer, as well as confidence levels of the M target category locking devices, where M is a natural number greater than or equal to 1; The filtering unit is configured to filter the M locking device features based on the confidence levels of the M target category locking devices to obtain filtered m locking device features, where m is a non-negative integer; The classification unit is configured to input the filtered m locking device features and the image features into the second reasoning layer when the value of m is not zero, to obtain a classification result of the target image output by the second reasoning layer.
2. A locking device detection method, characterized in that: include: Embedding a multi-label classification layer in any one of the fusion layers of the target detection model, wherein the target detection model includes N layers, where N is a natural number greater than or equal to 1, and the target detection model is trained based on sample images, multiple sample category locking devices, and classification result labels of the sample images; Convert the multi-label classification layer and the first K layers before the multi-label classification layer in the target detection model into a first inference layer, and convert the NK layers after the multi-label classification layer in the target detection model into a second inference layer, where K is a natural number greater than or equal to 1; Inputting a target image and M preset target category locking devices into the first reasoning layer, obtaining image features corresponding to the target image and M locking device features output by the first reasoning layer, as well as confidence levels of the M target category locking devices, where M is a natural number greater than or equal to 1; Based on the confidence levels of the M target category locking devices, the M locking device features are filtered to obtain m filtered locking device features, where m is a non-negative integer; When the value of m is not zero, the filtered m locking device features and the image features are input into the second reasoning layer to obtain a classification result of the target image output by the second reasoning layer.
3. The locking device detection method according to claim 2, characterized in that: Inputting the target image and the preset M target category locking devices into the first reasoning layer, obtaining the image features and M locking device features corresponding to the target image output by the first reasoning layer, as well as the confidence levels of the M target category locking devices, includes: Input the target image and the preset M target category locking devices into the first K layers, and obtain the image features corresponding to the target image and the M locking device features output by the first K layers; Based on the multi-label classification layer, the similarity between each locking device feature and the image feature of the corresponding area is calculated, and based on the similarity between each locking device feature and the image feature of the corresponding area, the category confidence of each locking device feature is calculated.
4. The locking device detection method according to claim 2, characterized in that: The filtering of the M locking device features based on the confidence levels of the M target category locking devices to obtain filtered m locking device features includes: Based on the confidence levels of the M target category locking devices and a preset threshold, the M locking device features are filtered to obtain the filtered m locking device features.
5. The locking device detection method according to claim 2, characterized in that: After obtaining the filtered m locking device features, the method further includes: When the value of m is zero, the target detection model directly outputs the classification result of the target image as empty.
6. The locking device detection method according to claim 2, characterized in that: The training process of the target detection model includes: Acquire a sample image and a plurality of sample category locking devices, and determine a classification result label of the sample image; Using the sample image and multiple sample category locking devices as training samples, and using the classification result labels of the sample image as sample labels to train an initial target detection model; Iteratively optimize the parameters of the initial target detection model to obtain the target detection model.
7. The locking device detection method according to claim 6, characterized in that: The initial target detection model includes a first initial inference layer and a second initial inference layer, the first initial inference layer includes an initial multi-label classification layer and an initial first K layer, the initial first K layer refers to the K layers in the initial target detection model located before the initial multi-label classification layer, and the initial target detection model includes N layers; Correspondingly, the training of the initial target detection model includes: Inputting the sample image and multiple sample category locking devices into the initial front K layers, and obtaining sample image features and multiple sample category locking device features corresponding to the sample image output by the initial front K layers; Based on the initial multi-label classification layer, calculating the similarity between each sample locking device feature and the sample image feature of the corresponding area; Calculating a loss function value based on the similarity between each sample locking device feature and the image feature of the corresponding area and the classification result label of the sample image; The initial multi-label classification layer is trained based on the loss function value.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the locking device detection method according to any one of claims 2 to 7 is implemented.
9. A non-transitory computer-readable storage medium, characterized in that The storage medium stores a computer software program, which, when executed by a processor, implements the locking device detection method according to any one of claims 2 to 7.
10. A computer program product, characterized in that The computer program product comprises a computer program, wherein when the computer program is executed by a processor, the locking device detection method according to any one of claims 2 to 7 is implemented.
