Multi-modal person wearing state self-adaptive detection method and device

Abstract

The invention discloses a multi-modal person wearing state adaptive detection method and device. The method comprises the following steps: acquiring an image of a target object; calling a pre-stored reference object image and a corresponding text description from a database; processing the image of the target object, the reference object image and the corresponding text description by using a multi-modal CLIP model to obtain a target image feature vector, a reference image feature vector and a reference text vector; respectively calculating the similarity between the target image feature vector and the character vector and the similarity between the reference image feature vector and the character vector; comparing the two similarities, and judging whether the user is in a preset wearing state or not according to a comparison result. According to the method, multi-modal information of images and texts is combined, image and text data processing is carried out based on the fine-tuned CLIP model, the understanding ability and accuracy of a recognition task are improved, and the method has high generalization ability and deployment efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of wearing state recognition, and in particular to a multi-modal personnel wearing state adaptive detection method and device. Background Art

[0002] In many industries, it is common for employees to wear specific work clothes. In high-risk working environments such as construction sites, petrochemical plants, and power companies, personnel who enter the protection area without wearing the correct work clothes will bring certain safety hazards and cause confusion in personnel management. In the medical industry, medical staff need to wear specific uniforms and protective equipment to ensure the hygiene and safety of the medical environment. Employees wear specific work clothes to meet the safety requirements and standards of various industries. Work clothes play an important role in preventing safety accidents. Therefore, monitoring whether work clothes are worn as required is a necessary measure for safe production.

[0003] Existing work clothes wearing methods based on image recognition usually adopt the framework of target detection, such as yolo, FasterR-CNN, etc., to realize real-time monitoring of video streams and timely alarms; some methods use convolutional neural network (CNN) technology to extract feature information of work clothes, such as color, pattern and logo, etc. This method uses image recognition technology to determine whether the target object meets the work clothes wearing requirements; some products will link the work clothes wearing detection system with the access control system, and only when it is detected that the personnel are wearing the work clothes correctly can enter the protection area.

[0004] Most of the existing detection methods are aimed at specific types of work clothes or work clothes of specified colors, and they do not have strong generalization. When encountering new work clothes category scenarios, it is necessary to continuously add data and conduct special model training for each type of work clothes. At present, most products on the market will continue to increase the workload due to the increase in the types of work clothes, and the detection efficiency is low. Secondly, some existing detection methods require pre-setting of similarity thresholds, which are too dependent on prior experience and rely only on single image features, ignoring the details of the work clothes, and have low accuracy. Summary of the invention

[0005] Technical purpose: In view of the above technical problems, the present invention proposes a multi-modal personnel wearing status adaptive detection method and device.

[0006] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution:

[0007] A multi-modal adaptive detection method for personnel wearing status comprises the following steps:

[0008] S100, acquiring an image of a target object, wherein the target object is any one of work clothes, work hats or work shoes worn by a person;

[0009] S200, retrieving a pre-stored reference object image and a corresponding text description from a database, wherein the reference object corresponds to the target object and is any one of a work uniform, a work cap or a work shoe in a standard wearing state;

[0010] S300, using a multimodal CLIP model to process the target object image, the reference object image and the corresponding text description to obtain a target image feature vector, a reference image feature vector and a reference text vector;

[0011] S400, calculating a first similarity score according to the target image feature vector and the reference text vector, and calculating a second similarity score according to the reference image feature vector and the reference text vector;

[0012] S500: Compare the first similarity score and the second similarity score, and determine whether the target object is in a predetermined wearing state according to the comparison result, and output the determination result.

[0013] Preferably, in step S100, acquiring an image of the target object includes the following steps:

[0014] S101. Set the position of the fixed-point camera to take photos, and take photos of the personnel wearing the clothes with the required clarity and completeness;

[0015] S102: Use a neural network model to detect and crop the captured photos, and output images of target objects that meet the conditions.

[0016] Preferably, in step S200, one target object corresponds to reference object images in three standard wearing states, namely, a front image, a back image and a side image, and each reference object image corresponds to a text description, which includes a description of the color or logo pattern of the reference object.

[0017] Preferably, in step S300, the multimodal CLIP model is pre-adjusted using a triplet loss function based on a dynamic margin mechanism, and the specific steps include:

[0018] A1. Collect images of reference objects and corresponding text descriptions to form a data set. Each data point in the data set contains an image and one or more related text descriptions.

[0019] A2. Construct triples. For each piece of training data, select a text description as an anchor point, the reference object image corresponding to the text description as a positive sample, and randomly select an image from the data set as a negative sample.

[0020] A3. Load the pre-trained CLIP model and use it as the base model.

[0021] A4. Construct a triplet loss function based on a dynamic margin mechanism:

[0022] Let a represent the embedding vector of the anchor point, p represent the embedding vector of the positive sample, and p represent the embedding vector of the negative sample;

[0023] A dynamic margin mechanism is introduced. The margin α is used to control the minimum distance difference between positive samples and negative samples. It is dynamically adjusted according to the training process. The margin α is defined as follows:

[0024]

[0025] Among them, t is the current training step, T is the total number of training steps, α0 is the initial margin, α f is the final margin;

[0026] The triplet loss function is defined as follows:

[0027]

[0028] Where d() represents the Euclidean distance metric function, N is the size of the training set, and i is the i-th training data in the training set.

[0029] A5. Minimize the triplet loss function by gradient descent method, update the model parameters, and obtain the fine-tuned CLIP model.

[0030] Preferably, the step S300 includes the following steps:

[0031] S301, inputting the target object image into the fine-tuned CLIP model to obtain the target image feature vector;

[0032] S302, inputting the reference object image into the fine-tuned CLIP model to obtain a reference image feature vector;

[0033] S303: Input the text description corresponding to the reference object image into the fine-tuned CLIP model to obtain a reference text vector.

[0034] Preferably, the step S400 includes the following steps:

[0035] S401, normalize the target image feature vector and the reference text vector respectively, and then calculate the similarity, and then obtain the maximum value of the similarity score to obtain a first similarity score Q1, the calculation method is as follows:

[0036] Q1 = max(dot(norm(F1),norm(F Li )), i∈[1,n]) (3)

[0037] Among them, F1 is the target image feature vector, n is the number of text descriptions, Li is the text description corresponding to the i-th reference object image, and F Li is the reference text vector corresponding to the text description Li, dot is the vector dot product operation, norm is the normalization operation, and max is the maximum value operation;

[0038] S402, normalize the reference image feature vector and the reference text vector respectively, and then calculate the similarity, and then obtain the average value of the similarity scores to obtain a second similarity score Q2, the calculation method is as follows:

[0039] Q2 = avg(dot(norm(F Ri ),norm(F Li )),i∈[1,n]) (4)

[0040] Among them, Ri is the i-th reference object image, F Ri is the reference image feature vector corresponding to the reference object image Ri, and avg is the averaging operation.

[0041] S403, compare Q1 and Q2, if Q1 is greater than or equal to Q2, the target object is in a predetermined target work clothes wearing state, if Q1 is less than Q2, the target object is in an unpredicted target work clothes wearing state:

[0042]

[0043] Among them, Flag is an indicator of whether the work clothes are in the predetermined target wearing state.

[0044] A multi-modal personnel wearing state adaptive detection device, comprising:

[0045] A first image acquisition module, used to acquire an image of a target object, wherein the target object is any one of work clothes, work hats or work shoes worn by a person;

[0046] A second image acquisition module is used to retrieve a pre-stored reference object image and a corresponding text description from a database, wherein the reference object corresponds to the target object and is any one of a work uniform, a work cap or a work shoe in a standard wearing state;

[0047] A multimodal CLIP model is used to process the target object image, the reference object image and the corresponding text description to obtain a target image feature vector, a reference image feature vector and a reference text vector;

[0048] A similarity calculation module, used to calculate a first similarity score based on a target image feature vector and a reference text vector, and to calculate a second similarity score based on a reference image feature vector and a reference text vector;

[0049] The comparison and output module compares the first similarity score and the second similarity score, and judges whether the target object is in a predetermined wearing state according to the comparison result and outputs the judgment result.

[0050] Beneficial effects: Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0051] The present invention combines the multimodal information of images and texts, processes image and text data based on a fine-tuned CLIP model, improves the comprehension and accuracy of recognition tasks, and has high generalization ability and deployment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The flowchart of the present invention for detecting whether the work clothes are worn correctly;

[0053] Figure 2a Flow chart of steps S31 to S33 of fine-tuning the multimodal CLIP model of the present invention;

[0054] Figure 2b Flow chart of steps S34 to S35 of fine-tuning the multimodal CLIP model of the present invention;

[0055] Figure 3 This is an example diagram of the detection result of judging whether the target image is worn correctly in the present invention. DETAILED DESCRIPTION

[0056] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0057] The present invention proposes a multi-modal personnel wearing state adaptive detection method based on dynamic triples, including the following process:

[0058] Determine a target object that meets the conditions from the image captured by the image acquisition device;

[0059] Obtaining the uploaded reference object image and the corresponding text description of the reference object from the backend system;

[0060] The multimodal CLIP model is fine-tuned and trained using a triplet loss function method based on a dynamic margin mechanism, and the image feature vector of the target object, the image feature vector of the reference object, and the text vector of the reference object are obtained using the fine-tuned multimodal CLIP model.

[0061] The target image feature vector and the reference text vector are similarly calculated to obtain a score Q1, and the reference image feature vector and the reference text vector are similarly calculated to obtain a score Q2;

[0062] Compare Q1 and Q2 to determine whether the target object is in the predetermined target work clothes wearing state.

[0063] The present invention has high generalization ability and higher deployment efficiency, and can be applied to the detection of work clothes, work hats, work shoes, etc., as well as the detection of the correct wearing of safety helmets, safety belts, etc. The present invention combines the multimodal information of images and texts for judgment. Compared with the method that only uses a single image information, the present invention supplements the information that is easily overlooked by vision, enhances the understanding ability of the recognition task, and thus improves the accuracy. In addition, the present invention does not rely on prior artificial experience to set the similarity comparison threshold, and performs comparison adaptively to more accurately detect the wearing status of work clothes.

[0064] Let's take work clothes as an example. Figures 1 to 3 , the working process and principle of the present invention are described in detail.

[0065] A multi-modal personnel wearing state adaptive detection method based on dynamic triples, comprising the following steps:

[0066] S1, determining a target object from an image acquired by an image acquisition device;

[0067] S2. Obtain the uploaded reference object image and the corresponding text description of the reference object from the background system;

[0068] S3, obtaining the image feature vector of the target object, the image feature vector of the reference object, and the text vector of the reference object through the multimodal CLIP model fine-tuned and trained by the triplet loss function based on the dynamic margin mechanism;

[0069] S4, calculating the similarity between the target image feature vector and the reference text vector to obtain a score Q1; calculating the similarity between the reference image feature vector and the reference text vector to obtain a score Q2; by comparing Q1 and Q2, determining whether the target object is in a predetermined wearing state.

[0070] In step S1, acquiring a target object from an image acquisition device comprises the following steps:

[0071] S11. Set the position of the fixed-point camera to ensure that the photos of the personnel wearing work clothes can be clearly taken;

[0072] S12. Use the detection model trained by the YOLO network to detect what people are wearing in the video photo, and determine the target object that meets the conditions, that is, the cropped image M1 of the clothes worn by the person.

[0073] In step S2, obtaining a reference image and a text description from a background system includes the following steps:

[0074] S21, the reference images are three types of images of the same work uniform, a front work uniform image R1, a back work uniform image R2, and a side work uniform image R3;

[0075] S22, describe the work clothes with reference to the text. You can describe it from the color, for example, the main color is red, and the sleeves are gray; you can also describe it from the logo pattern, for example, the text pattern printed with "xx company"; each type of image corresponds to a specific text description, L1, L2, L3. This step uses text information to dynamically register the work clothes, and enhances the ability to judge whether the work clothes are worn correctly by supplementing text information, thereby enhancing the characteristic distinguishability of the work clothes.

[0076] like Figure 2a and Figure 2b As shown, in step S3, fine-tuning the multimodal CLIP model includes the following steps:

[0077] S31. Prepare a dataset of image-text pairs: Collect a dataset containing work uniform images and corresponding text descriptions. Each data point contains an image and one or more related text descriptions to ensure that the model can understand the correlation between the image and the text during training.

[0078] S32. Construct triples: For each piece of training data, select a text description as an anchor, the work clothes image corresponding to the text description as a positive sample, and randomly select another image from the data set as a negative sample. The purpose of using triples as training data is to train the model to identify the correct image-text correspondence and better distinguish the incorrect correspondence. The anchor data plays a benchmark role in triple-based training. It helps determine the basis for comparison. By fixing the anchor and changing the positive and negative samples, the model can be effectively trained to understand the relationship between different inputs. This is crucial to improving the model's understanding ability, especially when dealing with complex and diverse data such as work clothes.

[0079] S33, Load CLIP model: Load the pre-trained Chinese version CLIP model and use it as the base model. The weights of the pre-trained Chinese version CLIP model have been trained on a large amount of data and can capture the general characteristics between images and texts. Using the pre-trained model as a basis can achieve better performance faster on limited data sets;

[0080] S34. Construct a triplet loss function based on a dynamic margin mechanism:

[0081] Let a represent the embedding vector of the anchor, p represent the embedding vector of the positive sample, and n represent the embedding vector of the negative sample.

[0082] At the same time, a dynamic margin mechanism is introduced. The margin α is used to control the minimum distance difference between positive samples and negative samples. It is dynamically adjusted according to the training process. A smaller margin is set at the beginning and gradually increases as the training progresses. It is defined as follows:

[0083]

[0084] Where t is the current training step, T is the total number of training steps, α0 is the initial margin, and α f is the final margin.

[0085] The triplet loss function is defined as follows:

[0086]

[0087] where d(·,·) represents the Euclidean distance metric function and N is the size of the training set.

[0088] The present invention designs a dynamic margin mechanism that allows the model to gradually improve its ability to distinguish correct and incorrect matches during training. A smaller margin at the beginning allows the model to converge more easily, and then gradually increase the margin as training progresses, forcing the model to learn to cluster correct pairs more closely together and push incorrect pairs further apart. This strategy helps avoid overfitting and promotes the generalization ability of the model.

[0089] S35, fine-tune training and update model:

[0090] The gradient descent method is used to minimize the loss function in formula (2), thereby updating the model parameters and obtaining the fine-tuned CLIP model. The training part trains the triples as a whole. In the inference stage, the triples are input into the model separately, and then the corresponding features are obtained for subsequent similarity calculation.

[0091] In step S4, determining whether the target object is in a predetermined target work clothes wearing state includes the following steps:

[0092] S41, in the inference stage, the target cropped image M1 is input into the fine-tuned CLIP model to obtain the image feature vector F1 of the target object;

[0093] S42, in the inference stage, the front work uniform image R1, the back work uniform image R2, and the side work uniform image R3 are respectively input into the fine-tuned CLIP model to obtain the image feature vector F of the reference object R1 , F R2 and F R3 ;

[0094] S43, in the inference stage, the description L1 corresponding to the front work clothes, the description L2 corresponding to the back work clothes, and the description L3 corresponding to the side work clothes are respectively input into the fine-tuned CLIP model to obtain the text vector F of the reference object L1 , F L2 and F L3 ;

[0095] S44, normalize the target image feature vector and the reference text vector respectively, and then calculate the similarity, and then obtain the maximum value of the similarity score to obtain Q1. The calculation method is as follows:

[0096] Q1=max(dot(norm(F1),norm(F Li ))) (3)

[0097] Among them, i is the number of text descriptions, which is a positive integer, where i = 1, 2, 3, dot is a vector dot product operation, norm is a normalization operation, and max is a maximum value operation.

[0098] S45, normalize the reference image feature vector and the reference text vector respectively and perform similarity calculation, then obtain the average value of the similarity scores to obtain Q2, the calculation method is as follows:

[0099] Q2=avg(dot(norm(F Ri ),norm(F Li )), i∈n) (4)

[0100] Among them, n is 3, dot is the vector dot product operation, norm is the normalization operation, and avg is the average operation.

[0101] S46, compare Q1 and Q2. If Q1 is greater than or equal to Q2, the target object is in a predetermined target work clothes wearing state. If Q1 is less than Q2, the target object is in an unpredicted target work clothes wearing state, which is expressed as follows:

[0102]

[0103] Among them, Flag is an indicator of whether the work clothes are in the predetermined target wearing state.

[0104] Figure 3 This is an example diagram of the detection results of using the method of the present invention to determine whether relevant personnel are wearing work clothes correctly.

[0105] Compared with the previous methods, the method of the present invention can adapt to various work clothes, dynamically register work clothes reference images and text, and is more generalized. Some previous methods use the specified type of target detection work clothes to achieve this. This method is not suitable for each scene with specific work clothes, and each work clothes with different characteristics requires image acquisition, training and other links. The algorithm does not have the generalization ability of multiple scenes, which increases the workload.

[0106] The present invention does not rely on the target detection network to identify specific work clothes for judgment. The multimodal CLIP model fine-tuned after collecting data has generalization and can be applied to different project scenarios without the need to repeatedly retrain the model; based on the comparison of three reference images and corresponding reference text information with the target image, it no longer relies on the need to collect a large amount of material to add to the model training every time. Therefore, the present invention has higher generalization ability and higher deployment efficiency.

[0107] In terms of detection effect, the previous methods rely on the comparison of feature vectors between the pre-set target detection results and the reference image, which is very dependent on the accuracy of the target detection results and is greatly affected by the environment. The method of the present invention matches image information and text information from multiple angles. Compared with using only single image information, it supplements the information that is easily overlooked by vision, and further improves the understanding ability of the recognition task through multi-modal capabilities, thereby improving accuracy.

[0108] In addition, some methods rely on prior manual experience to set thresholds for comparison. They rely on manual prior threshold setting, and need to adjust thresholds repeatedly for experiments in different project scenarios. They only use image features, and have low accuracy. However, this method calculates the similarity between the target image feature vector and the reference image feature vector and the text vector, and compares them adaptively, which greatly improves the accuracy.

[0109] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A multi-modal adaptive detection method for wearing status of a person, characterized in that: Includes steps: S100, acquiring an image of a target object, wherein the target object is any one of work clothes, work hats or work shoes worn by a person; S200, retrieving a pre-stored reference object image and a corresponding text description from a database, wherein the reference object corresponds to the target object and is any one of a work uniform, a work cap or a work shoe in a standard wearing state; S300, using a multimodal CLIP model to process the target object image, the reference object image and the corresponding text description to obtain a target image feature vector, a reference image feature vector and a reference text vector; S400, calculating a first similarity score according to the target image feature vector and the reference text vector, and calculating a second similarity score according to the reference image feature vector and the reference text vector; S500: Compare the first similarity score and the second similarity score, and determine whether the target object is in a predetermined wearing state according to the comparison result, and output the determination result.

2. A multi-modal adaptive detection method for wearing status of a person according to claim 1, characterized in that: In step S100, acquiring an image of the target object includes the following steps: S101. Set the position of the fixed-point camera to take photos, and take photos of the personnel wearing the clothes with the required clarity and completeness; S102: Use a neural network model to detect and crop the captured photos, and output images of target objects that meet the conditions.

3. A multi-modal adaptive detection method for wearing status of a person according to claim 1, characterized in that: In step S200, one target object corresponds to reference object images in three standard wearing states, namely, a front image, a back image and a side image, and each reference object image corresponds to a text description, which includes a description of the color or logo pattern of the reference object.

4. A multi-modal adaptive detection method for wearing status of a person according to claim 1, characterized in that: In step S300, the multimodal CLIP model is pre-adjusted using a triplet loss function based on a dynamic margin mechanism, and the specific steps include: A1. Collect images of reference objects and corresponding text descriptions to form a data set. Each data point in the data set contains a reference object image and multiple text descriptions. A2. Construct triples. For each piece of training data, select a text description as an anchor point, the reference object image corresponding to the text description as a positive sample, and randomly select an image from the data set as a negative sample. A3. Load the pre-trained CLIP model and use it as the base model. A4. Construct a triplet loss function based on a dynamic margin mechanism: Let a represent the embedding vector of the anchor point, p represent the embedding vector of the positive sample, and p represent the embedding vector of the negative sample; A dynamic margin mechanism is introduced. The margin α is used to control the minimum distance difference between positive samples and negative samples. It is dynamically adjusted according to the training process. The margin α is defined as follows: Among them, t is the current training step, T is the total number of training steps, α0 is the initial margin, α f is the final margin; The triplet loss function is defined as follows: Where d() represents the Euclidean distance metric function, N is the size of the training set, and i is the i-th training data in the training set; A5. Minimize the triplet loss function by gradient descent method, update the model parameters, and obtain the fine-tuned CLIP model.

5. A multi-modal adaptive detection method for wearing status of a person according to claim 4, characterized in that: The step S300 includes the following steps: S301, inputting the target object image into the fine-tuned CLIP model to obtain the target image feature vector; S302, inputting the reference object image into the fine-tuned CLIP model to obtain a reference image feature vector; S303: Input the text description corresponding to the reference object image into the fine-tuned CLIP model to obtain a reference text vector.

6. A multi-modal adaptive detection method for wearing status of a person according to claim 1, characterized in that: The step S400 includes the following steps: S401, normalize the target image feature vector and the reference text vector respectively, and then calculate the similarity, and then obtain the maximum value of the similarity score to obtain a first similarity score Q1, the calculation method is as follows: Q1 =max(dot(norm(F1),norm(F Li )),i∈[1,n]) (3) Among them, F1 is the target image feature vector, n is the number of text descriptions, Li is the text description corresponding to the i-th reference object image, and F Li is the reference text vector corresponding to the text description Li, dot is the vector dot product operation, norm is the normalization operation, and max is the maximum value operation; S402, normalize the reference image feature vector and the reference text vector respectively, and then calculate the similarity, and then obtain the average value of the similarity scores to obtain a second similarity score Q2, the calculation method is as follows: Q2 =avg(dot(norm(F Ri ),norm(F Li )),i∈[1,n]) (4) Among them, Ri is the i-th reference object image, F Ri is the reference image feature vector corresponding to the reference object image Ri, and avg is the averaging operation; S403, compare Q1 and Q2, if Q1 is greater than or equal to Q2, the target object is in a predetermined target work clothes wearing state, if Q1 is less than Q2, the target object is in an unpredicted target work clothes wearing state: Among them, Flag is an indicator of whether the work clothes are in the predetermined target wearing state.

7. A multi-modal adaptive detection device for wearing status of a person, characterized in that: include: A first image acquisition module, used to acquire an image of a target object, wherein the target object is any one of work clothes, work hats or work shoes worn by a person; A second image acquisition module is used to retrieve a pre-stored reference object image and a corresponding text description from a database, wherein the reference object corresponds to the target object and is any one of a work uniform, a work cap or a work shoe in a standard wearing state; A multimodal CLIP model is used to process the target object image, the reference object image and the corresponding text description to obtain a target image feature vector, a reference image feature vector and a reference text vector; A similarity calculation module, used to calculate a first similarity score based on a target image feature vector and a reference text vector, and to calculate a second similarity score based on a reference image feature vector and a reference text vector; The comparison and output module compares the first similarity score and the second similarity score, and judges whether the target object is in a predetermined wearing state according to the comparison result and outputs the judgment result.

Images