Tattoo scar recognition method, system, device and medium

Abstract

The present invention provides a tattoo scar recognition method, system, device and medium, which relates to the field of image intelligent recognition technology, including: obtaining tattoo and scar images, inputting the images and tattoo recognition prompt words into a VQA multimodal large model, respectively obtaining the features of the image and the features of the tattoo recognition prompt words through the large model; aligning and fusing the features of the image and the features of the tattoo recognition prompt words to generate a tattoo recognition result; fusing the tattoo recognition result with the scar recognition prompt words to generate a new scar recognition prompt word, inputting the new scar recognition prompt word and the image into the VQA multimodal large model, respectively obtaining the features of the image and the features of the new scar recognition prompt word through the large model; aligning and fusing the features of the image and the features of the new scar recognition prompt word to generate a scar recognition result. The present invention can accurately recognize tattoos and scars at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent image recognition, and in particular to a tattoo scar recognition method, system, equipment and medium. Background Art

[0002] With the rapid development of biometric technology, identity authentication methods based on individual biometrics have been widely used in many fields such as security monitoring, border control and personal device protection. Identification methods and technologies based on primary biometrics (such as face and fingerprint) have made great progress, but it is difficult for a single biometric to fully meet the actual application needs in open scenarios. For example, in large-scale crowd monitoring scenarios, it is challenging to obtain high-quality biometric data due to factors such as distance, light, and occlusion. In addition, biometrics may be unavailable due to damage or deliberate destruction. In applications such as intelligent video surveillance, the explosive growth of video data has put forward higher requirements for the efficiency and accuracy of identity recognition. Traditional identity recognition relies on primary biometrics such as face, fingerprint and iris, but in non-controllable scenarios and non-cooperative user conditions, identity recognition systems based on primary biometrics often fail due to the inability to obtain the required biometric data; at this time, auxiliary biometrics can provide important auxiliary functions for identity recognition.

[0003] Tattoos and scars are important auxiliary biometric features and are of great value in assisting identity recognition, especially when data such as faces and fingerprints cannot be obtained or are damaged by humans. In this context, auxiliary biometric features such as tattoos and scars have gradually become an important supplement in the field of identity recognition due to their uniqueness and stability. Tattoos, as a biological feature with a long history, not only have a place in culture and personal expression, but also play a key role in rapid identity recognition in emergency situations. For example, in disaster victim identification and criminal investigation, accurate extraction and matching of tattoo information is crucial for rapid positioning and confirmation of identity.

[0004] However, the development of tattoo recognition technology still faces many challenges. Automatic retrieval of tattoo images needs to solve problems including image quality, angle deviation, illumination change and complex background interference. In addition, existing tattoo search methods mostly focus on matching cropped tattoo images, which is far from the actual application demand for automatic detection and recognition of tattoos in original images. Realizing fully automatic robust large-scale tattoo scar image retrieval is still a challenging problem.

[0005] In order to improve the accuracy and robustness of tattoo recognition, researchers have begun to explore the application of deep learning technology in tattoo recognition. Deep learning methods are expected to overcome the limitations of traditional manual feature design and achieve more accurate tattoo detection and matching by automatically learning high-level features of images. Nevertheless, how to design a unified framework that can handle tattoo detection and feature learning at the same time, and how to quantify model output to improve recognition efficiency, are still hot topics in current research. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a tattoo and scar recognition method, system, device and medium to achieve the purpose of accurately identifying tattoos and scars at the same time.

[0007] The embodiment of the present application provides the following technical solution: a tattoo scar recognition method, comprising:

[0008] Acquire tattoo and scar images, input the tattoo and scar images and tattoo recognition prompt words into a VQA multimodal large model, and obtain features of the tattoo and scar images and features of the tattoo recognition prompt words through the VQA multimodal large model;

[0009] Aligning and fusing the features of the tattoo and scar images and the features of the tattoo recognition prompt words, and generating a tattoo recognition result based on the fused features;

[0010] The tattoo recognition result is merged with the scar recognition prompt word to generate a new scar recognition prompt word, the new scar recognition prompt word and the tattoo and scar images are input into the VQA multimodal large model, and the features of the tattoo and scar images and the features of the new scar recognition prompt word are respectively obtained through the VQA multimodal large model;

[0011] The features of the tattoo and scar images and the features of the new scar recognition prompt words are aligned and fused, and a scar recognition result is generated based on the fused features.

[0012] According to an embodiment of the present application, it includes:

[0013] The features of the tattoo and scar images and the features of the tattoo identification prompt word are aligned through the projection layer; and the features of the tattoo and scar images and the features of the new scar identification prompt word are aligned.

[0014] According to an embodiment of the present application, aligning the features of the tattoo and scar images and the features of the tattoo recognition prompt word through a projection layer; and aligning the features of the tattoo and scar images and the features of the new scar recognition prompt word, including:

[0015] The MLP neural network is used to transform features of different dimensions into the same dimension using linear transformation to achieve feature alignment.

[0016] According to an embodiment of the present application, the VQA multimodal large model adopts cross entropy loss, and the cross entropy loss function L is:

[0017]

[0018] Among them, M is a categorical variable; y ic is a sign function. If the category of sample i is equal to c, then y is 1, otherwise y is 0; p ic is the predicted probability that the observed sample i belongs to category c.

[0019] According to an embodiment of the present application, the VQA multimodal large model adopts a transformer model architecture.

[0020] According to an embodiment of the present application, after acquiring the tattoo and scar images, the method further includes:

[0021] The tattoo and scar images are subjected to data preprocessing, data annotation, data segmentation, data enhancement and data standardization.

[0022] The present application also provides a tattoo scar recognition system, comprising:

[0023] A first feature extraction module is used to obtain tattoo and scar images, input the tattoo and scar images and tattoo recognition prompt words into a multimodal VQA multimodal large model, and obtain features of the tattoo and scar images and features of the tattoo recognition prompt words through the VQA multimodal large model;

[0024] A tattoo recognition module, used to align and fuse the features of the tattoo and scar images and the features of the tattoo recognition prompt words, and generate a tattoo recognition result according to the fused features;

[0025] a second feature extraction module, configured to fuse the tattoo recognition result with the wound recognition prompt word to generate a new wound recognition prompt word, input the new wound recognition prompt word and the tattoo and wound images into the VQA multimodal large model, and obtain the features of the tattoo and wound images and the features of the new wound recognition prompt word through the VQA multimodal large model;

[0026] The scar recognition module is used to align and fuse the features of the tattoo and scar images and the features of the new scar recognition prompt words, and generate a scar recognition result based on the fused features.

[0027] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned tattoo scar recognition method when executing the computer program.

[0028] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the above-mentioned tattoo scar recognition method.

[0029] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above technical solutions adopted in the embodiments of this specification include at least: the embodiments of the present invention propose a method and system for simultaneously performing tattoo scar detection and learning based on the original image in a real scene, which are as follows:

[0030] 1. The embodiment of the present invention adopts the VQA method to realize the recognition of tattoos and scars; this method fine-tunes the Lora layer under the QwenVL framework to realize the recognition of tattoos and scars;

[0031] 2. The embodiment of the present invention can not only accurately identify whether there are tattoos or scars in the image task, but also accurately locate the part of the body where the tattoos or scars are located, such as on the face, neck, arm or palm; 3. The embodiment of the present invention innovatively realizes batch image synchronous detection and identification of tattoos and scars. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a schematic diagram of a tattoo scar recognition method flow chart according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of the structure of a tattoo scar recognition system according to an embodiment of the present invention;

[0035] Figure 3 It is a structural schematic diagram of the computer device of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0037] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0038] like Figure 1 As shown, an embodiment of the present invention provides a tattoo scar recognition method, comprising:

[0039] S101. Acquire tattoo and scar images, input the tattoo and scar images and tattoo recognition prompt words into a VQA multimodal large model, and obtain features of the tattoo and scar images and features of the tattoo recognition prompt words through the VQA multimodal large model;

[0040] S102. Aligning and fusing the features of the tattoo and scar images and the features of the tattoo recognition prompt words, and generating a tattoo recognition result based on the fused features;

[0041] S103. The tattoo recognition result is merged with the scar recognition prompt word to generate a new scar recognition prompt word, and the new scar recognition prompt word and the tattoo and scar images are input into the VQA multimodal large model, and the features of the tattoo and scar images and the features of the new scar recognition prompt word are respectively obtained through the VQA multimodal large model;

[0042] S104. Align and fuse the features of the tattoo and scar images and the features of the new scar recognition prompt word, and generate a scar recognition result based on the fused features.

[0043] In this embodiment, the method includes: aligning the features of the tattoo and scar images and the features of the tattoo identification prompt word through a projection layer; and aligning the features of the tattoo and scar images and the features of the new scar identification prompt word through a projection layer.

[0044] In the specific implementation, the image features are aligned with the prompt word features, and the MLP neural network is used to transform the features of different dimensions into the same dimension using linear transformation. The process of linear transformation is to project the single modality into the same multimodal representation space using a function. The formula is:

[0045] X m =f(X1,X2,……,X n )

[0046] In this embodiment, after aligning the image features with the prompt word features, an association matrix between the image and the prompt information is calculated. In the association matrix, the larger the value corresponding to the image with tattoos and scars, the greater the value corresponding to the image. The alignment of the image and the prompt word features can be left-aligned, right-aligned, or centered.

[0047] In this embodiment, the cross entropy loss is used in the VQA multimodal large model, and the cross entropy loss function L is:

[0048]

[0049] Among them, M is a categorical variable; y ic is a sign function (0 or 1), if the category of sample i is equal to c, then y is 1, otherwise y is 0; p ic is the predicted probability that the observed sample i belongs to category c.

[0050] In specific implementation, the VQA multimodal large model adopts the transformer model architecture. In practical applications, the attention mechanism calculation expression in the transformer model architecture is as follows:

[0051]

[0052] In this embodiment, after acquiring the tattoo and scar images, the following steps are further included:

[0053] The tattoo and scar images are subjected to data preprocessing, data annotation, data segmentation, data enhancement and data standardization.

[0054] In specific implementation, the implementation process of this embodiment specifically includes the following steps:

[0055] Data preparation stage: The training data mainly includes five parts. The first part comes from the public tattoo scar dataset, the second part is the tattoo scar images collected from the Internet, the third part is the data taken with stickers, etc., the fourth part is a part of the data drawn manually, and the fifth part is the data augmented by image enhancement methods (grayscale transformation, random cropping, random splicing, etc.) on the existing data. Due to the limitations of the public dataset and the needs of business evaluation, a considerable amount of data was artificially created and augmented for model training and evaluation.

[0056] The general steps for preparing data for multimodal large model training are:

[0057] Step 1: Collect data. Collect data of various modalities, such as text data, image data, etc. Depending on the specific requirements of the model, specific datasets may be required, such as image-text pairs.

[0058] Step 2: Data preprocessing: Preprocess the data of each modality, including but not limited to image scaling and normalization, text segmentation and encoding, etc.

[0059] Step 3: Data labeling. If the model requires supervised learning, the data needs to be labeled. Labeling may include labels for text classification, similarity labels for image and text matching, etc.

[0060] Step 4: Data segmentation: Split the dataset into training set, validation set, and test set to facilitate model training and evaluation.

[0061] Step 5: Data enhancement: In order to improve the generalization ability of the model, it may be necessary to enhance the data, such as image rotation, flipping, scaling, etc.

[0062] Step 6: Data standardization: Ensure that data from different modalities are consistent in value range and distribution to facilitate model training.

[0063] Multimodal large model: The embodiment of the present invention is based on the QwenVL multimodal large model. The application of multimodal large models in the field of visual question answering (VQA) is a cutting-edge and active research topic. The VQA task requires the model to understand natural language questions and give accurate answers based on the image content. This requires not only the model to have strong visual perception capabilities, but also language understanding and reasoning capabilities. The research and application of multimodal large models in the field of VQA are developing rapidly. By combining computer vision and natural language processing technologies, these models can better understand and answer questions about images.

[0064] Model training: The architecture of the VQA model, which may include an encoder, an attention mechanism, a fusion layer, and an output layer. For VQA tasks, multi-label cross entropy loss is usually used because each question may have multiple correct answers. The optimization algorithm SGD is used to train the model. During the training process, techniques such as gradient accumulation and learning rate decay are required.

[0065] Model quantization: Model quantization is a technique for deep learning model optimization that aims to reduce the model's memory footprint and speed up the inference process while maintaining the model's accuracy as much as possible. This technology is particularly important for deploying deep learning models to resource-constrained environments (such as mobile devices and embedded systems). A key challenge of quantization is to maintain the model's accuracy as much as possible while reducing the model size and computing requirements. This requires a trade-off between model design, training, and quantization strategies. In some cases, the model architecture may need to be adjusted to accommodate the loss of accuracy caused by quantization.

[0066] System deployment: This method uses a triton-based model deployment solution. Model deployment is the process of applying a trained machine learning model to a real application. This process involves multiple steps, including model optimization, conversion, testing, and running in a production environment.

[0067] like Figure 2 As shown, the present application also provides a tattoo scar recognition system 200, comprising:

[0068] A first feature extraction module 201 is used to obtain tattoo and scar images, input the tattoo and scar images and tattoo recognition prompt words into a multimodal VQA multimodal large model, and obtain features of the tattoo and scar images and features of the tattoo recognition prompt words through the VQA multimodal large model;

[0069] The tattoo recognition module 202 is used to align and fuse the features of the tattoo and scar images and the features of the tattoo recognition prompt words, and generate a tattoo recognition result according to the fused features;

[0070] The second feature extraction module 203 is used to fuse the tattoo recognition result with the wound recognition prompt word to generate a new wound recognition prompt word, input the new wound recognition prompt word and the tattoo and wound images into the VQA multimodal large model, and obtain the features of the tattoo and wound images and the features of the new wound recognition prompt word through the VQA multimodal large model;

[0071] The scar recognition module 204 is used to align and fuse the features of the tattoo and scar images and the features of the new scar recognition prompt word, and generate a scar recognition result according to the fused features.

[0072] In a specific implementation, the tattoo recognition module 202 aligns the features of the tattoo and scar images and the features of the tattoo recognition prompt word through the projection layer; the scar recognition module 204 aligns the features of the tattoo and scar images and the features of the new scar recognition prompt word through the projection layer. The tattoo recognition module 202 and the scar recognition module 204 use an MLP neural network to convert features of different dimensions into the same dimension using linear transformation to achieve feature alignment.

[0073] In a specific implementation, after the first feature extraction module 201 acquires the tattoo and scar images, the method further includes: performing data preprocessing, data labeling, data segmentation, data enhancement and data standardization on the tattoo and scar images.

[0074] In one embodiment, a computer device is provided, such as Figure 3 As shown, it includes a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor. When the processor 302 executes the computer program, the above-mentioned tattoo scar recognition method is implemented.

[0075] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0076] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing the above-mentioned tattoo scar recognition method.

[0077] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0078] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0079] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A tattoo scar recognition method, characterized in that: include: Acquire tattoo and scar images, input the tattoo and scar images and tattoo recognition prompt words into a VQA multimodal large model, and obtain features of the tattoo and scar images and features of the tattoo recognition prompt words through the VQA multimodal large model; Aligning and fusing the features of the tattoo and scar images and the features of the tattoo recognition prompt words, and generating a tattoo recognition result based on the fused features; The tattoo recognition result is merged with the scar recognition prompt word to generate a new scar recognition prompt word, the new scar recognition prompt word and the tattoo and scar images are input into the VQA multimodal large model, and the features of the tattoo and scar images and the features of the new scar recognition prompt word are respectively obtained through the VQA multimodal large model; The features of the tattoo and scar images and the features of the new scar recognition prompt words are aligned and fused, and a scar recognition result is generated based on the fused features.

2. The tattoo scar recognition method according to claim 1, characterized in that: include: Aligning the features of the tattoo and scar images and the features of the tattoo recognition prompt words through a projection layer; Furthermore, the features of the tattoo and scar images and the features of the new scar identification prompt word are aligned.

3. The tattoo scar recognition method according to claim 2, characterized in that: Aligning the features of the tattoo and scar images and the features of the tattoo recognition prompt words through a projection layer; Furthermore, aligning the features of the tattoo and scar images and the features of the new scar identification prompt word includes: The MLP neural network is used to transform features of different dimensions into the same dimension using linear transformation to achieve feature alignment.

4. The tattoo scar recognition method according to claim 1, characterized in that: The VQA multimodal large model uses cross entropy loss, and the cross entropy loss function L is: Among them, M is a categorical variable; y ic is a sign function. If the category of sample i is equal to c, then y is 1, otherwise y is 0; p ic is the predicted probability that the observed sample i belongs to category c.

5. The tattoo scar recognition method according to claim 1, characterized in that: The VQA multimodal large model adopts the transformer model architecture.

6. The tattoo scar recognition method according to claim 1, characterized in that: After obtaining tattoo and scar images, it also includes: The tattoo and scar images are subjected to data preprocessing, data annotation, data segmentation, data enhancement and data standardization.

7. A tattoo scar recognition system, characterized in that: include: A first feature extraction module is used to obtain tattoo and scar images, input the tattoo and scar images and tattoo recognition prompt words into a multimodal VQA multimodal large model, and obtain features of the tattoo and scar images and features of the tattoo recognition prompt words through the VQA multimodal large model; A tattoo recognition module, used to align and fuse the features of the tattoo and scar images and the features of the tattoo recognition prompt words, and generate a tattoo recognition result according to the fused features; a second feature extraction module, configured to fuse the tattoo recognition result with the wound recognition prompt word to generate a new wound recognition prompt word, input the new wound recognition prompt word and the tattoo and wound images into the VQA multimodal large model, and obtain the features of the tattoo and wound images and the features of the new wound recognition prompt word through the VQA multimodal large model; The scar recognition module is used to align and fuse the features of the tattoo and scar images and the features of the new scar recognition prompt words, and generate a scar recognition result based on the fused features.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the tattoo scar recognition method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the tattoo scar recognition method according to any one of claims 1 to 6.

Images