Operating personnel cross-lens tracking method, system and equipment and storage medium
By combining text features and image features, the problem of poor cross-lens recognition and tracking effects of workers in power grid construction environments is solved, and higher recognition accuracy and tracking effects are achieved.
Patent Information
- Application Number
- CN202411454127.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2025-02-07
AI Technical Summary
The existing cross-lens identification and tracking technology of operators is not effective in the power grid construction environment, especially when the operators wear similar or the same clothes, it is difficult to accurately identify and track, affecting the safe production and stable operation of the power grid.
Using a multimodal data processing method combining text features and image features, the operator's description text and image field images of each lens are obtained through the pre-trained operator's cross-lens tracking model, and the text features and image features are extracted, and the The feature fusion module obtains more discriminant fusion features, ultimately achieving accurate cross-lens tracking for operators.
By adding text data, the unchanging characteristics of the operator can be better described and the accuracy of the identification results can be improved. Especially when the operators wear similar or the same clothes, the accuracy of cross-lens tracking is significantly improved.
Smart Images
Figure CN119399690A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of machine recognition and relates to a method, system, device and storage medium for tracking operators across lenses. Background Art
[0002] Cross-lens operator recognition and tracking technology is a method to match the same person in different lenses. Most of the current cross-lens operator recognition and tracking technologies are based on optimization or deep learning. Most of these technologies are based on a strong assumption that the same operator wears the same clothes in different lenses, and different operators wear different clothes in the same lens. Therefore, when the same operator wears different clothes in different lenses, and when different operators wear similar or the same clothes in different lenses, the existing cross-lens operator recognition and tracking technology does not perform well.
[0003] In the face of cross-lens recognition and tracking of power grid workers, due to the relatively complex power grid construction environment, direct manual recognition and tracking of power grid workers is not only inefficient but also extremely costly. At the same time, since the clothing of power grid workers is relatively uniform, it is often difficult to accurately identify and track power grid workers using existing cross-lens recognition and tracking technologies for workers, which may pose a threat to the safe production and stable operation of the power grid. To address this problem, many cross-lens recognition and tracking technologies for workers with changed clothing have been proposed. These technologies mainly improve the accuracy of recognition and tracking by eliminating the negative impact of clothing features and extracting more discriminative features.
[0004] At present, the cross-shot identification and tracking technology of workers based on clothing changes can be divided into occlusion-based methods and soft biometric information-based methods. Among them, the occlusion-based method uses a mask to make the network ignore clothing information to obtain a representation that is unrelated to clothing. This method relies on the accuracy of the mask to a certain extent, and directly ignores the clothing part, which will cause the network to learn an incomplete representation. The method based on soft biometric information aims to use biometric information that is more discriminative than clothing information to help complete the cross-shot identification and tracking of workers. This method generally adopts a dual-branch structure in the training stage. The main branch is responsible for completing the cross-shot identification and tracking technology of personnel, and the sub-branch predicts soft biometric information, such as shape, contour and gait, and then uses a certain method to let the soft biometric information predicted by the sub-branch guide the main branch to extract more discriminative soft biometric information features. The biggest challenge of this method is how to obtain accurate biometric information. Although the deep learning-based method currently used can extract some biometric information features, it is inaccurate and incomplete, and the cost of manually collecting biometric information labels is huge. In general, although these methods can improve accuracy to a certain extent, they all ignore the relationship between clothing and worker identification and tracking. This results in the current cross-lens worker identification and tracking technology having poor effects in scenarios where different workers wear similar or identical clothing. Summary of the invention
[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, system, device and storage medium for tracking operators across lenses.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] In a first aspect, the present invention provides a method for tracking workers across lenses, comprising: obtaining a description text of the workers and images of the work site of each lens; obtaining worker tracking results of the work site images of each lens through a pre-trained worker cross-lens tracking model according to the description text of the workers and the images of the work site of each lens; wherein the worker cross-lens tracking model comprises a text feature extraction module, an image feature extraction module, a feature fusion module and a prediction module; the text feature extraction module is used to extract text features of the description text; the image feature extraction module is used to extract image features of the work site images of each lens; the feature fusion module is used to fuse the text features and the image features to obtain fused features; the prediction module is used to obtain the worker tracking results of the work site images of each lens according to the fused features.
[0008] Optionally, the text feature extraction module is constructed using a pre-trained language model.
[0009] Optionally, the image feature extraction module includes a semantic segmentation model module, an image encoder module and an image feature fusion module; the semantic segmentation model module is used to divide the workers in each lens operation site image into upper and lower parts according to preset upper weights and lower weights through a pre-trained semantic segmentation model to obtain a semantic segmentation map, and then multiply the semantic segmentation map with the corresponding lens operation site image to obtain a part weight awareness image; the image encoder module is used to encode each lens operation site image to obtain a first image feature, and to encode the part weight awareness image to obtain a second image feature; the image feature fusion module is used to fuse the first image feature and the second image feature to obtain the image feature of each lens operation site image.
[0010] Optionally, the weight of the upper part is greater than the weight of the lower part; when the first image feature and the second image feature are fused, the fusion is performed based on the cosine similarity of the first image feature and the second image feature.
[0011] Optionally, the feature fusion module includes a text-image fusion module, an image-text fusion module and an overlay module; the text-image fusion module is used to take the text feature as the main feature and the image feature as the secondary feature, and to fuse the text feature and the image feature in a cross-attention manner to obtain a first fused feature; the image-text fusion module is used to take the text feature as the secondary feature and the image feature as the main feature, and to fuse the text feature and the image feature in a cross-attention manner to obtain a second fused feature; the overlay module is used to weightedly overlay the first fused feature and the second fused feature according to preset first fusion feature weights and second fusion feature weights to obtain a fused feature.
[0012] Optionally, the prediction module is constructed using a fully connected layer.
[0013] According to a second aspect of the present invention, a system for tracking workers across shots is provided, comprising: a data acquisition module for acquiring description text of workers and images of work sites of each shot; an identification module for obtaining worker tracking results of work sites of each shot based on the description text of workers and images of work sites of each shot through a pre-trained cross-shot tracking model for workers; wherein the cross-shot tracking model for workers comprises a text feature extraction module, an image feature extraction module, a feature fusion module and a prediction module; the text feature extraction module is used to extract text features of the description text; the image feature extraction module is used to extract image features of the work site images of each shot; the feature fusion module is used to fuse the text features and the image features to obtain fused features; the prediction module is used to obtain worker tracking results of work site images of each shot based on the fused features.
[0014] Optionally, the image feature extraction module includes a semantic segmentation model module, an image encoder module and an image feature fusion module; the semantic segmentation model module is used to divide each lens operation site image into upper and lower parts according to preset upper part weights and lower part weights through a pre-trained semantic segmentation model to obtain a semantic segmentation map, and multiply the semantic segmentation map with each lens operation site image to obtain a part weight awareness image; the image encoder module is used to encode each lens operation site image to obtain a first image feature, and encode the part weight awareness image to obtain a second image feature; the image feature fusion module is used to fuse the first image feature and the second image feature to obtain the image of each lens operation site image features; the feature fusion module includes a text-image fusion module, an image-text fusion module and a superposition module; the text-image fusion module is used to take the text feature as the main feature, the image feature as the secondary feature, and fuse the text feature and the image feature in a cross-attention manner to obtain a first fused feature; the image-text fusion module is used to take the text feature as the secondary feature, the image feature as the main feature, and fuse the text feature and the image feature in a cross-attention manner to obtain a second fused feature; the superposition module is used to weightedly superimpose the first fused feature and the second fused feature according to the preset first fused feature weight and the second fused feature weight to obtain a fused feature.
[0015] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for tracking workers across shots when executing the computer program.
[0016] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for tracking workers across shots are implemented.
[0017] Compared with the prior art, the present invention has the following beneficial effects:
[0018] The cross-shot tracking method for workers of the present invention obtains description text of workers and work site images of each lens, and then uses a pre-trained cross-shot tracking model for workers to extract text features of the description text based on a text feature extraction module, extract image features of the work site images of each lens based on an image feature extraction module, and then fuse the text features and the image features through a feature fusion module to obtain fused features, and finally obtains the worker tracking results of the work site images of each lens according to the fused features through a prediction module. Because text data can better describe the invariant features of workers than image data, description text is used as multimodal data, and more discriminative features are extracted from both image streams and text streams. For situations where workers wear similar or identical clothes, the accuracy of the recognition results can be effectively improved based on the more discriminative fused features, and the cross-shot tracking and tracking tasks of workers can be better completed. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 The present invention is a flowchart of a method for tracking workers across shots according to an embodiment of the present invention.
[0020] Figure 2 This is a structural block diagram of a model for tracking operators across shots according to an embodiment of the present invention.
[0021] Figure 3 It is a structural block diagram of a system for tracking workers across shots according to an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0024] The present invention is further described in detail below in conjunction with the accompanying drawings:
[0025] See also Figure 1 In one embodiment of the present invention, a method for tracking workers across lenses is provided, which can accurately complete the tracking and tracking tasks of workers across lenses when the workers are wearing uniform clothes.
[0026] Specifically, the operator cross-shot tracking method of the present invention comprises the following steps:
[0027] S1: Obtain the description text of the operator and the work site images of each lens.
[0028] S2: Based on the description text of the operator and the work site images of each lens, the operator tracking results of the work site images of each lens are obtained through the pre-trained operator cross-lens tracking model.
[0029] Among them, the cross-lens tracking model for operators includes a text feature extraction module, an image feature extraction module, a feature fusion module and a prediction module; the text feature extraction module is used to extract the text features of the description text; the image feature extraction module is used to extract the image features of the work site images of each lens; the feature fusion module is used to fuse the text features and the image features to obtain fused features; the prediction module is used to obtain the operator tracking results of the work site images of each lens according to the fused features.
[0030] The description text is an overall description of the operator, which may include facial feature description, body shape description, hairstyle description, etc. The operator tracking result of each lens operation scene image is the probability that the operator of each lens operation scene image is the same person.
[0031] Generally speaking, when the visual system determines whether multiple images of people are the same person, it first determines the facial features of the person, followed by features such as body shape and hairstyle, and finally clothing features. Therefore, the features used to determine whether a person is the same can be divided into three parts: deep features, intermediate features, and shallow features. To solve the problem of cross-lens tracking and tracking of operators in power scenarios, we must try our best to extract deep features, and use intermediate features and shallow features for auxiliary judgment.
[0032] The current cross-lens tracking and tracking technology for operators mainly uses RGB images (RGB images are the most common type of digital images, which create colors based on the sensitivity of the human eye to different colors), thermal infrared images and near-infrared images as multimodal data. Different from this, the present invention introduces text data as multimodal data. Text data can directly express specific information and meanings, such as facts, opinions and emotions. Compared with image data, the semantics of text data is clearer and easier to parse and understand. Therefore, text data can better and more clearly describe the more discriminative features of operators. Therefore, the present invention adds text data to RGB image data and uses text data as another modality to extract more discriminative features.
[0033] The cross-shot tracking method for workers of the present invention obtains description text of workers and work site images of each lens, and then uses a pre-trained cross-shot tracking model for workers to extract text features of the description text based on a text feature extraction module, extract image features of the work site images of each lens based on an image feature extraction module, and then fuse the text features and the image features through a feature fusion module to obtain fused features, and finally obtains the worker tracking results of the work site images of each lens according to the fused features through a prediction module. Because text data can better describe the invariant features of workers than image data, description text is used as multimodal data, and more discriminative features are extracted from both image streams and text streams. For situations where workers wear similar or identical clothes, the accuracy of the recognition results can be effectively improved based on the more discriminative fused features, and the cross-shot tracking and tracking tasks of workers can be better completed.
[0034] In a possible implementation, the text feature extraction module is constructed using a pre-trained language model.
[0035] Specifically, the text feature extraction module is required to be able to capture the subtle differences and complex features in the descriptive text, so that the extracted text features are richer and more meaningful. Therefore, using a pre-trained language model as a text feature extraction module can not only extract rich text information, but also increase the versatility of the present invention.
[0036] Pre-trained language models can use BERT, GPT-2, XLNet and T5, etc. BERT, GPT-2, XLNet and T5 are all pre-trained language models based on the Transformer architecture, which are widely used in the field of natural language processing. The description text of the operator is sent to the text feature extraction module, and the text features are obtained after being processed by the text feature extraction module.
[0037] In one possible implementation, see Figure 2The image feature extraction module includes a semantic segmentation model module, an image encoder module and an image feature fusion module.
[0038] Among them, the semantic segmentation model module is used to divide the workers in each lens operation site image into upper and lower parts according to the preset upper part weight and lower part weight, through the pre-trained semantic segmentation model to obtain a semantic segmentation map, and then multiply the semantic segmentation map with the corresponding each lens operation site image to obtain a part weight awareness image; the image encoder module is used to encode each lens operation site image to obtain a first image feature, and encode the part weight awareness image to obtain a second image feature; the image feature fusion module is used to fuse the first image feature and the second image feature to obtain the image feature of the said each lens operation site image.
[0039] Specifically, the image feature extraction module is used to extract the features of the workers in the image from the input image. Since the power workers wear similar or identical clothes, there will be few more discriminative features in the clothing, but that does not mean there are none. Therefore, in order to extract more discriminative features, that is, deep-level features, the present invention introduces a pre-trained semantic segmentation model. Unlike the previous three-part form of dividing the workers into the upper part, the middle part and the lower part, in this embodiment, the workers are divided into two parts, the upper part and the lower part. Among them, the upper part is the head of the worker, and the lower part is the part below the head of the worker.
[0040] At the same time, since the lower part reflects the clothing of the operator to a large extent, but also contains middle-level features such as posture and gait, the lower part information cannot be directly and roughly ignored. However, in order to reflect the difference between the upper and lower parts, the upper and lower weights are introduced, so that when extracting more discriminative features, those features with insufficient discriminative power are not ignored, because identifying operators is an overall evaluation. At the same time, using a part-based semantic segmentation model, the accuracy of the semantic segmentation results affects the final performance to a certain extent. At the same time, adding the semantic segmentation results also affects the model's perception of the overall character of the operator. Therefore, in this embodiment, the original work site images of each lens are also subjected to feature extraction.
[0041] The image encoder module can use the classic Vision Transformer (ViT) architecture as an image encoder to complete the task of extracting features. Among them, the ViT architecture is an innovative method that applies the Transformer model widely used in natural language processing to the field of computer vision. The ViT architecture achieves effective extraction of image features by dividing the image into a series of small blocks and treating these small blocks as words in the Transformer model.
[0042] In a possible implementation manner, the weight of the upper part is greater than the weight of the lower part.
[0043] Specifically, in order to make the model focus more on features other than clothing, the upper weight is designed to be a, the lower weight is designed to be 1-a, and a soft constraint is imposed on the upper weight, requiring a>0.5, to ensure that the network pays more attention to the deep features of people, so as to extract more discriminative features.
[0044] Optionally, when fusing the first image feature and the second image feature, the fusion is performed based on cosine similarity between the first image feature and the second image feature.
[0045] Specifically, based on the cosine similarity of the first image feature and the second image feature, if the cosine similarity of the first image feature and the second image feature vector is very similar in some dimensions, the value of one of the image features in these dimensions can be selected, and the value of the other image feature can be selected in other dimensions, or interpolation can be performed based on the cosine similarity, and finally the first image feature and the second image feature can be fused. This can not only ensure the integrity and diversity of image features, but also eliminate redundant information and save memory space.
[0046] In a possible implementation, the main processing process of the image feature extraction module is as follows:
[0047] Step 1: Send the images of each lens operation scene to the image encoder module to obtain the first image feature.
[0048] Step 2: Send the images of each lens operation scene to the pre-trained semantic segmentation model to obtain a semantic segmentation map, which divides the person into two parts: the upper part and the lower part. Then, multiply the images of each lens operation scene with the obtained semantic segmentation map to obtain a part weight awareness image, as shown in the following formula:
[0049] I part =I input *S input
[0050] Among them, I part is the part weight awareness image, I input is the working scene image of each lens, S input is a semantic segmentation map. Then the part weight consciousness image is sent to the image encoder to obtain the second image feature.
[0051] Step three: sending the first image feature and the second image feature to a fusion module, and fusing the first image feature and the second image feature based on the cosine similarity of the first image feature and the second image feature to obtain the image features of the image of each lens operation scene.
[0052] In a possible implementation, the feature fusion module includes a text-image fusion module, an image-text fusion module and an overlay module; the text-image fusion module is used to use the text feature as the main feature and the image feature as the secondary feature, and to fuse the text feature and the image feature in a cross-attention manner to obtain a first fused feature; the image-text fusion module is used to use the text feature as the secondary feature and the image feature as the main feature, and to fuse the text feature and the image feature in a cross-attention manner to obtain a second fused feature; the overlay module is used to weightedly overlay the first fused feature and the second fused feature according to a preset first fused feature weight and a preset second fused feature weight to obtain a fused feature.
[0053] Specifically, the feature fusion module first fuses the image features from the image feature extraction module and the text features from the text feature extraction module, and then obtains the prediction result based on the fused features.
[0054] Multimodal feature fusion is one of the research focuses of multimodal tasks. Some use simple fusion strategies, such as addition and concatenation, to save computing resources. Some use complex modules, such as layer-by-layer fusion, to obtain better representation features after fusion, but it requires a huge amount of calculation. In addition, when fusing text features and image features, considering the different characteristics of different inputs, there may be three possibilities when predicting the results: the prediction results can be basically obtained through image features, the prediction results can be basically obtained through text features, and the results are obtained through the combined action of text features and image features. Therefore, it is not possible to effectively determine which of the text features and image features is the main feature and which is the secondary feature in a certain prediction.
[0055] At the same time, when using the cross-attention method to fuse features of different modalities, the features of different modalities should be divided into primary features and secondary features. The secondary features are to provide useful information to further improve the prediction accuracy. However, in actual situations, it is possible that image features provide most of the discriminant features, and text features are just supplementary; it is also possible that text features provide most of the discriminant features, and image features are just supplementary. Therefore, once the primary and secondary features are fixed during prediction, they cannot be fused according to the actual situation of the input data.
[0056] Therefore, compared with most methods that directly fuse data from all modalities at the same time, and because text features and image features are inherently difficult to fuse, this implementation proposes a new fusion strategy in which image features and text features are used as primary features and secondary features, respectively, so that two different feature fusion results can be obtained. Then, a learnable parameter, namely the first fusion feature weight and the second fusion feature weight, is used to allow the model to judge the importance of image features and text features to the prediction results, thereby allowing the model to adaptively determine the fusion method of image features and text features.
[0057] In a possible implementation, the main processing process of the feature fusion module is as follows:
[0058] Step 1: Taking the text feature as the main feature and the image feature as the secondary feature, the text feature and the image feature are fused by cross-attention to obtain a first fused feature.
[0059] Step 2: Taking the text feature as a secondary feature and the image feature as a primary feature, the text feature and the image feature are fused by a cross-attention method to obtain a second fused feature.
[0060] Step 3: Then, according to the weight of the first fusion feature and the weight of the second fusion feature, in this implementation, the weight of the first fusion feature is b, and the weight of the second fusion feature is 1-b, and then the first fusion feature and the second fusion feature are weighted and superimposed to obtain the fusion feature, as shown in the following formula:
[0061] f fusion =b*f image-text +(1-b)*f text-image
[0062] Among them, f fusion is the fusion feature, f text-image is the first fusion feature, f image-text It is the second fusion feature.
[0063] In a possible implementation, the prediction module is constructed using a fully connected layer.
[0064] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.
[0065] See also Figure 3 In yet another embodiment of the present invention, a system for tracking workers across shots is provided, which can be used to implement the above-mentioned method for tracking workers across shots. Specifically, the system for tracking workers across shots includes a data acquisition module and an identification module.
[0066] Among them, the data acquisition module is used to obtain the description text of the operator and the work site images of each lens; the recognition module is used to obtain the operator tracking results of the work site images of each lens based on the description text of the operator and the work site images of each lens through the pre-trained operator cross-lens tracking model.
[0067] Among them, the cross-lens tracking model for operators includes a text feature extraction module, an image feature extraction module, a feature fusion module and a prediction module; the text feature extraction module is used to extract the text features of the description text; the image feature extraction module is used to extract the image features of the work site images of each lens; the feature fusion module is used to fuse the text features and the image features to obtain fused features; the prediction module is used to obtain the operator tracking results of the work site images of each lens according to the fused features.
[0068] In a possible implementation, the image feature extraction module includes a semantic segmentation model module, an image encoder module and an image feature fusion module; the semantic segmentation model module is used to divide each lens operation scene image into upper and lower parts according to a preset upper part weight and lower part weight through a pre-trained semantic segmentation model to obtain a semantic segmentation map, and multiply the semantic segmentation map with each lens operation scene image to obtain a part weight awareness image; the image encoder module is used to encode each lens operation scene image to obtain a first image feature, and encode the part weight awareness image to obtain a second image feature; the image feature fusion module is used to fuse the first image feature and the second image feature to obtain the image of each lens operation scene; The image features of the scene image; the feature fusion module includes a text-image fusion module, an image-text fusion module and an overlay module; the text-image fusion module is used to take the text feature as the main feature, the image feature as the secondary feature, and fuse the text feature and the image feature in a cross-attention manner to obtain a first fused feature; the image-text fusion module is used to take the text feature as the secondary feature, the image feature as the main feature, and fuse the text feature and the image feature in a cross-attention manner to obtain a second fused feature; the overlay module is used to weightedly overlay the first fused feature and the second fused feature according to the preset first fused feature weight and the second fused feature weight to obtain a fused feature.
[0069] All relevant contents of each step involved in the embodiment of the aforementioned operator cross-shot tracking method can be referred to the functional description of the functional modules corresponding to the operator cross-shot tracking system in the embodiment of the present invention, and will not be repeated here.
[0070] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.
[0071] In another embodiment of the present invention, a computer device is provided, the computer device comprising a processor and a memory, the memory being used to store a computer program, the computer program comprising program instructions, and the processor being used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the cross-lens tracking method for operators.
[0072] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the cross-shot tracking method for operators in the above embodiment.
[0073] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for tracking a worker across shots, characterized in that: include: Obtain the description text of the operator and the work site images of each lens; According to the description text of the operator and the work scene images of each lens, the operator tracking results of the work scene images of each lens are obtained through the pre-trained operator cross-lens tracking model; The operator cross-lens tracking model includes a text feature extraction module, an image feature extraction module, a feature fusion module and a prediction module; the text feature extraction module is used to extract the text features of the description text; the image feature extraction module is used to extract the image features of the images of the operation scene of each lens; the feature The fusion module is used to fuse the text features and the image features to obtain fusion features; the prediction module is used to obtain the operator tracking results of the work site images of each lens according to the fusion features.
2. The method for tracking workers across shots according to claim 1, characterized in that: The text feature extraction module is constructed using a pre-trained language model.
3. The method for tracking workers across shots according to claim 1, characterized in that: The image feature extraction module includes a semantic segmentation model module, an image encoder module and an image feature fusion module; The semantic segmentation model module is used to divide the workers in each lens operation scene image into upper and lower parts according to the preset upper part weight and lower part weight by using the pre-trained semantic segmentation model to obtain a semantic segmentation map, and then multiply the semantic segmentation map with the corresponding lens operation scene image to obtain a part weight awareness image; The image encoder module is used to encode the image of each lens operation scene to obtain a first image feature, and to encode the part weight awareness image to obtain a second image feature; The image feature fusion module is used to fuse the first image feature and the second image feature to obtain the image features of the images of the operation scene of each lens.
4. The method for tracking workers across shots according to claim 3, characterized in that: The weight of the upper part is greater than the weight of the lower part; when the first image feature and the second image feature are fused, the fusion is performed based on the cosine similarity of the first image feature and the second image feature.
5. The method for tracking workers across shots according to claim 1, characterized in that: The feature fusion module includes a text-image fusion module, an image-text fusion module and a superposition module; The text-image fusion module is used to use the text feature as the main feature and the image feature as the secondary feature, and fuse the text feature and the image feature in a cross-attention manner to obtain a first fusion feature; The image-text fusion module is used to use the text feature as a secondary feature and the image feature as a primary feature, and fuse the text feature and the image feature in a cross-attention manner to obtain a second fused feature; The superposition module is used for weighted superposition of the first fusion feature and the second fusion feature to obtain a fusion feature according to the preset first fusion feature weight and the second fusion feature weight.
6. The method for tracking workers across shots according to claim 1, characterized in that: The prediction module is constructed using a fully connected layer.
7. A system for tracking workers across shots, characterized in that: include: A data acquisition module is used to obtain the description text of the operator and the operation site images of each lens; The recognition module is used to obtain the operator tracking results of each lens operation scene image according to the operator description text and each lens operation scene image through the pre-trained operator cross-lens tracking model; The operator cross-lens tracking model includes a text feature extraction module, an image feature extraction module, a feature fusion module and a prediction module; the text feature extraction module is used to extract the text features of the description text; the image feature extraction module is used to extract the image features of the images of the operation scene of each lens; the feature The fusion module is used to fuse the text features and the image features to obtain fusion features; the prediction module is used to obtain the operator tracking results of the work site images of each lens according to the fusion features.
8. The operator cross-shot tracking system according to claim 7, characterized in that: The image feature extraction module includes a semantic segmentation model module, an image encoder module and an image feature fusion module; The semantic segmentation model module is used to divide each lens operation scene image into upper and lower parts according to the preset upper part weight and lower part weight through the pre-trained semantic segmentation model to obtain a semantic segmentation map, and multiply the semantic segmentation map with each lens operation scene image to obtain a part weight awareness image; the image encoder module is used to encode each lens operation scene image to obtain a first image feature, and encode the part weight awareness image to obtain a second image feature; the image feature fusion module is used to fuse the first image feature and the second image feature to obtain the image feature of each lens operation scene image; The feature fusion module includes a text-image fusion module, an image-text fusion module and a superposition module; The text-image fusion module is used to use the text feature as the main feature and the image feature as the secondary feature, and fuse the text feature and the image feature in a cross-attention manner to obtain a first fused feature; the image-text fusion module is used to use the text feature as the secondary feature and the image feature as the main feature, and fuse the text feature and the image feature in a cross-attention manner to obtain a second fused feature; The superposition module is used for weighted superposition of the first fusion feature and the second fusion feature to obtain a fusion feature according to the preset first fusion feature weight and the second fusion feature weight.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for tracking workers across shots as described in any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for tracking workers across shots as described in any one of claims 1 to 6 are implemented.