Shooting and posture recognition method, device, equipment and storage medium

Abstract

The present application provides a shooting and posture recognition method, device, equipment and storage medium, the method comprising: inputting a clothing image into a preset appearance extraction network to obtain a clothing feature vector set; inputting a real-time image into a preset YOLO model to obtain a target person's intercepted image, the preset YOLO model is trained by the clothing feature vector set; performing posture analysis on the target person's intercepted image by a preset MMpose posture recognition model, generating posture key points and key point coordinates, and generating a posture feature vector set according to the posture key points and key point coordinates; inputting the clothing feature vector set and the posture feature vector set into a parameter matching model, determining a first shooting parameter according to the clothing feature vector set, and determining a second shooting parameter according to the posture feature vector set; determining a shooting time by using the posture feature vector set and a preset target feature vector set; at the shooting time, shooting the target person according to the first shooting parameter and the second shooting parameter.

Description

Technical Field

[0001] The present application relates to the field of automated camera technology, and in particular to a shooting and gesture recognition method, device, equipment and storage medium. Background Art

[0002] At present, automated shooting systems have been applied in the fashion industry, capturing the dynamic effects of clothing displays through posture estimation technology. As an advanced open source posture estimation toolbox, MMPose integrates multiple algorithms, can perform efficient 2D posture estimation, and supports key point detection and skeleton construction in the automated shooting process. However, due to the unconventionality of fashion styles, existing technologies still have limitations in complex posture recognition, variable lighting and background processing, low robustness and inaccurate shooting parameter settings, and the images captured by automated shooting do not meet the standards. Summary of the invention

[0003] The present application provides a shooting and gesture recognition method, device, equipment and storage medium for improving the shooting quality of images.

[0004] In a first aspect, an embodiment of the present application provides a shooting and gesture recognition method, the method comprising:

[0005] Input the clothing image into a preset appearance extraction network to extract features and obtain a clothing feature vector set;

[0006] Inputting the real-time image captured by the camera device into a preset YOLO model, locating the target person in the real-time image, and obtaining a captured image of the target person, wherein the preset YOLO model is trained by the clothing feature vector set;

[0007] Performing posture analysis on the captured image of the target person through a preset MMpose posture recognition model to generate posture key points and key point coordinates, and generating a posture feature vector set according to the posture key points and the key point coordinates;

[0008] Inputting the clothing feature vector set and the posture feature vector set into a preset parameter matching model, determining a first shooting parameter according to the clothing feature vector set, and determining a second shooting parameter according to the posture feature vector set;

[0009] The moment when the similarity between the posture feature vector set and the preset target feature vector set is greater than a preset similarity threshold is used as the shooting moment;

[0010] At the shooting moment, the target person is shot according to the first shooting parameter and the second shooting parameter.

[0011] In a second aspect, an embodiment of the present application provides a shooting and gesture recognition device, the device comprising:

[0012] A clothing recognition module is used to input clothing images into a preset appearance extraction network to perform feature extraction and obtain a clothing feature vector set;

[0013] A person positioning module, used for inputting the real-time image captured by the camera device into a preset YOLO model, positioning the target person in the real-time image, and obtaining a captured image of the target person, wherein the preset YOLO model is trained by the clothing feature vector set;

[0014] A posture recognition module performs posture analysis on the captured image of the target person through a preset MMpose posture recognition model, generates posture key points and key point coordinates, and generates a posture feature vector set according to the posture key points and the key point coordinates;

[0015] A parameter determination module, used for inputting the clothing feature vector set and the posture feature vector set into a preset parameter matching model, determining a first shooting parameter according to the clothing feature vector set, and determining a second shooting parameter according to the posture feature vector set;

[0016] A moment determination module, used for taking the moment when the similarity between the posture feature vector set and the preset target feature vector set is greater than a preset similarity threshold as the shooting moment;

[0017] A shooting control module is used to shoot the target person according to the first shooting parameter and the second shooting parameter at the shooting moment.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor;

[0019] The memory is used to store computer programs;

[0020] The processor is used to execute the computer program and implement the shooting and gesture recognition method as described in any one of the embodiments of the present application when executing the computer program.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor enables the processor to implement the shooting and gesture recognition method as described in any one of the embodiments of the present application.

[0022] The embodiment of the present application provides a shooting and posture recognition method, which includes: inputting a clothing image into a preset appearance extraction network to perform feature extraction and obtain a clothing feature vector set; inputting a real-time image captured by a camera device into a preset YOLO model to locate a target person in the real-time image and obtain a captured image of the target person, wherein the preset YOLO model is trained by the clothing feature vector set; performing posture analysis on the captured image of the target person by a preset MMpose posture recognition model to generate posture key points and key point coordinates, and generating a posture feature vector set according to the posture key points and key point coordinates; inputting the clothing feature vector set and the posture feature vector set into a preset parameter matching model, determining a first shooting parameter according to the clothing feature vector set, and determining a second shooting parameter according to the posture feature vector set; taking a moment when the similarity between the posture feature vector set and the preset target feature vector set is greater than a preset similarity threshold as a shooting moment; at the shooting moment, shooting the target person according to the first shooting parameter and the second shooting parameter. Through the above method, the clothing feature vector set is extracted in advance, which not only provides a reference for the subsequent model, but also does not occupy real-time computing power. The YOLO model is used to locate the person according to the clothing feature vector set, so that the target person can be accurately and quickly located and tracked in a complex background, and the judgment influence of non-target people in the fashion show scene is used. After the above preliminary positioning, the posture characteristics of the target person are carefully detected through the posture recognition model. Finally, the shooting parameters are matched according to the clothing characteristics and posture characteristics, and the target shooting parameters are determined from the historical images and the corresponding shooting parameters. When the target person makes a preset posture, the shooting is performed according to the target shooting parameters. In this way, automatic image capture is realized, and high-quality image capture is completed in a complex background. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0024] Figure 1 A schematic flow chart of a shooting and gesture recognition method provided in an embodiment of the present application;

[0025] Figure 2 A schematic block diagram of a photographing and gesture recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0030] See also Figure 1 , Figure 1 The embodiment of the present application also provides a schematic flow chart of a shooting and gesture recognition method. Figure 1 As shown, the specific steps of the shooting and gesture recognition method include: S101-S106.

[0031] S101, inputting a clothing image into a preset appearance extraction network to perform feature extraction to obtain a clothing feature vector set.

[0032] Exemplarily, the preset appearance extraction network is trained by historical clothing data, which includes clothing images and image annotations, and the image annotations include color annotations, color layout annotations, pattern annotations, accessory annotations, and strong reflective area annotations. The appearance extraction network is trained by clothing images and image annotations, and the appearance extraction network can be a YOLO model.

[0033] It should be noted that in this application, the identification and tracking of the target person is divided into three stages. Different models may be used in different stages, and the models used may only use part of the network layer. Some of the models used are online, and the other part is offline.

[0034] In the first stage, since clothing in the fashion industry usually includes specially set appearance features, such as special fashion decorative elements, it is easy to cause failure in character positioning, or a high positioning error rate in complex backgrounds. By identifying these appearance features, the target person corresponding to the clothing can be quickly located in a complex background. The target person is usually a model showing the clothing, and the appearance features of the clothing are used to match the preferred shooting parameters. For example, different color layouts present different shooting effects under different shooting parameters. When shooting, the shooting parameters need to be adjusted according to the above appearance features.

[0035] The first stage can be carried out in advance by inputting the prepared clothing images into the trained appearance extraction network to obtain a set of clothing feature vectors. In this way, real-time computing power will not be occupied during the fashion show, alleviating the pressure on computing resources.

[0036] S102, inputting the real-time image captured by the camera device into a preset YOLO model, locating the target person in the real-time image, and obtaining a captured image of the target person, wherein the preset YOLO model is trained by a clothing feature vector set.

[0037] Exemplarily, in the second stage, in the scene of tracking and locating the target person, the background may also include numerous non-target persons, which are serious interference sources and affect the robustness and recognition speed of model recognition. In order to solve the above problems, the present application performs auxiliary positioning through the extracted clothing features, and at the same time, reduces the computational complexity through the YOLO model to achieve preliminary positioning of the target person. After locating the target person, the YOLO model generates a positioning frame, and the image in the positioning frame is a captured image of the target person. The YOLO model has relatively low requirements for computing resources, and is suitable for achieving preliminary positioning of the target person, so as to quickly locate the target person in a complex background.

[0038] The real-time image is segmented through the positioning frame of the YOLO model to obtain the target person’s captured image and the background image. Subsequently, only the target person’s captured image input value needs to be input into the MMpose-based posture recognition model, which simplifies the input content and reduces the robustness and recognition speed of the model recognition.

[0039] S103, performing posture analysis on the captured image of the target person through a preset MMpose posture recognition model, generating posture key points and key point coordinates, and generating a posture feature vector set according to the posture key points and key point coordinates.

[0040] Exemplarily, in the third stage, the posture of the target person needs to be accurately identified so as to achieve posture tracking of the target person.

[0041] The pose analysis of the captured image of the target person is performed through the pose recognition model based on MMpose. The accuracy of key point positioning is improved by the importance of the geometric structure of the captured image of the target person through the category-free pose estimation method (CAPE), and a pose feature vector set is generated.

[0042] Since different postures require different shooting parameters, such as focal length parameters and shooting angle parameters, when implementing automated shooting, guiding parameter setting through posture features can improve the shooting quality of the image.

[0043] S104: Input the clothing feature vector set and the posture feature vector set into a preset parameter matching model, determine a first shooting parameter according to the clothing feature vector set, and determine a second shooting parameter according to the posture feature vector set.

[0044] Exemplarily, the clothing feature vector set is used to match exposure parameters, aperture parameters and sensitivity parameters, which are strongly related to the appearance characteristics of clothing. The posture feature vector set is used to match focal length parameters, shooting angle parameters and dynamic range parameters to reduce the occlusion effect caused by posture.

[0045] Through clothing features and posture features, the shooting parameters can be determined more accurately, that is, the appearance features of the clothing and the movement features of the posing posture are taken into account to reduce occlusion.

[0046] S105 , taking the moment when the similarity between the posture feature vector set and the preset target feature vector set is greater than a preset similarity threshold as the shooting moment.

[0047] Exemplarily, when the target person is detected to make a predetermined gesture, it is necessary to take a photo in time and use this moment as the shooting moment. The preset target feature vector set is generated according to the target gesture input by the user, and the similarity comparison process can be completed in the gesture recognition model. Specifically, the user inputs the target gesture into the gesture recognition model, and the gesture recognition model generates a preset target feature vector set according to the target gesture, and then performs a similarity comparison between the gesture feature vector set and the preset target feature vector set, and finally determines the shooting moment.

[0048] S106: At the shooting time, shoot the target person according to the first shooting parameter and the second shooting parameter.

[0049] The embodiment of the present application provides a shooting and posture recognition method, which includes: inputting a clothing image into a preset appearance extraction network to perform feature extraction and obtain a clothing feature vector set; inputting a real-time image captured by a camera device into a preset YOLO model to locate a target person in the real-time image and obtain a captured image of the target person, wherein the preset YOLO model is trained by the clothing feature vector set; performing posture analysis on the captured image of the target person by a preset MMpose posture recognition model to generate posture key points and key point coordinates, and generating a posture feature vector set according to the posture key points and key point coordinates; inputting the clothing feature vector set and the posture feature vector set into a preset parameter matching model, determining a first shooting parameter according to the clothing feature vector set, and determining a second shooting parameter according to the posture feature vector set; taking a moment when the similarity between the posture feature vector set and the preset target feature vector set is greater than a preset similarity threshold as a shooting moment; at the shooting moment, shooting the target person according to the first shooting parameter and the second shooting parameter. Through the above method, the clothing feature vector set is extracted in advance, which not only provides a reference for the subsequent model, but also does not occupy real-time computing power. The YOLO model is used to locate the person according to the clothing feature vector set, so that the target person can be accurately and quickly located and tracked in a complex background, and the judgment influence of non-target people in the fashion show scene is used. After the above preliminary positioning, the posture characteristics of the target person are carefully detected through the posture recognition model. Finally, the shooting parameters are matched according to the clothing characteristics and posture characteristics, and the target shooting parameters are determined from the historical images and the corresponding shooting parameters. When the target person makes a preset posture, the shooting is performed according to the target shooting parameters. In this way, automatic image capture is realized, and high-quality image capture is completed in a complex background.

[0050] In order to more clearly introduce the technical solution of the present application, the technical solution of the present application will be introduced through specific embodiments below. It should be noted that the specific embodiments are used to expand the technical solution of the present application, but are not intended to limit the present application.

[0051] In some embodiments, the preset appearance extraction network includes: a ResNet network, a conversion network and a feedforward neural network. The clothing image is input into the preset appearance extraction network to perform feature extraction to obtain a clothing feature vector set. The specific steps include: S1011-S1018.

[0052] S1011. Input the clothing image into the ResNet network to obtain a first residual map, a second residual map, and a third residual map.

[0053] S1012: Perform a deformable convolution process on the third residual image to obtain a deformable fourth residual image.

[0054] For example, Deformable Convolution is an improved convolution operation that aims to enhance the adaptability of convolutional neural networks (CNNs) to complex geometric shapes and deformations. Standard convolution operations use fixed geometric structures and sampling points at each position, and this fixed sampling method may not work well when processing images with complex deformations (such as fashion decorative elements). Deformable convolution introduces a learnable offset so that the sampling points of the convolution kernel can be dynamically adjusted, thereby capturing features in the input image more flexibly and adaptively.

[0055] S1013. Perform spatial attention processing on the deformable fourth residual image to obtain a spatial attention residual image.

[0056] Exemplarily, the deformable fourth residual map is subjected to average pooling and maximum pooling to obtain two one-dimensional residual maps. The two one-dimensional residual maps are concatenated to obtain a two-dimensional residual map. The two-dimensional residual map is convolved with a 7x7 convolution kernel to obtain a one-dimensional spatial weight block. The one-dimensional spatial weight block is subjected to a sigmoid function to obtain a compressed spatial weight block. The deformable fourth residual map and the compressed spatial weight block are subjected to element-by-element multiplication to obtain a spatial attention residual map.

[0057] S1014. Perform channel attention processing on the spatial attention residual map to obtain an improved residual map.

[0058] In some embodiments, channel attention processing is performed on the spatial attention residual map to obtain an improved residual map, including: performing average pooling and maximum pooling processing on the spatial attention residual map to obtain a pooled feature set; performing one-dimensional convolution processing and full connection processing on the pooled feature set to obtain a mapping feature set; performing element-by-element addition processing on the mapping feature set to obtain a cumulative feature set; performing sigmoid activation function processing on the cumulative feature set to obtain a one-dimensional channel attention weight block; performing element-by-element multiplication processing on the spatial attention residual map and the one-dimensional channel attention weight block to obtain an improved residual map.

[0059] In this way, the model can automatically learn which spatial locations in the image contain important information (for example, the location of fashion decorative elements) and give higher weights to these spatial locations. This can help the model better focus on key areas and ignore irrelevant background areas, thereby improving detection accuracy.

[0060] S1015, performing a first convolution process on the first residual image, the second residual image, and the deformable fourth residual image, respectively, to obtain a first feature set, a second feature set, and a third feature set.

[0061] S1016, performing a second convolution process on the improved residual image to obtain a fourth feature block. Combining the first feature block, the second feature block, the third feature block and the fourth feature block to obtain a multi-scale clothing residual image.

[0062] S1017, inputting the multi-scale clothing residual map into the conversion network to obtain enhanced representation information, inputting the enhanced representation information into the feedforward neural network to obtain a clothing feature vector set, the clothing feature vector set includes: a category vector and a position prediction vector.

[0063] Exemplarily, the conversion network includes a Transformer-based encoder and decoder, and the multi-scale clothing residual is input into the encoder, and the encoder processes the input features through a self-attention mechanism and a feedforward network. The decoder uses the encoder output and the query embedding and processes them through a cross-attention mechanism. Through the iteration of multi-layer encoder-decoder, enhanced representation information is obtained. In this way, the geometric features in the clothing features can be better focused.

[0064] In some embodiments, the enhanced representation information is input into a feedforward neural network to obtain a clothing feature vector set, the clothing feature vector set includes: a category vector and a position prediction vector, including: processing the enhanced representation information through a fully connected layer of the feedforward neural network to obtain a fully connected output feature. Performing activation function processing on the fully connected output feature to obtain an activated feature set. Performing classification head processing on the activated feature set to obtain a category vector. Performing regression head processing on the activated feature set to obtain a position prediction vector, including bounding box coordinates.

[0065] Through the above process, not only can the characteristics of fashion decorative elements be read and interpreted, but the accuracy of prediction results can also be improved in category identification and position positioning, providing strong technical support for fashion design and analysis.

[0066] In some embodiments, the real-time image captured by the camera device is input into a preset YOLO model, the target person in the real-time image is located, and a captured image of the target person is obtained. The preset YOLO model is trained by a clothing feature vector set. The specific steps include: S1021-S1026.

[0067] S1021. Perform feature extraction on the real-time image through multiple convolutional layers, where the functions of each convolutional layer include: convolution operation, batch normalization and LeakyReLU activation function, to obtain a first feature vector set.

[0068] S1022. Downsample the first feature vector set through a pooling layer to generate second feature vector sets of multiple scales.

[0069] S1023. Perform channel dimension reduction on the second feature vector set of each scale using 1x1 convolution, and then extract features using 3x3 convolution to obtain a third feature vector set.

[0070] S1024. Generate a prediction result using 1x1 convolution according to the third feature vector set. The prediction result includes: a candidate box, a confidence level, and a category probability.

[0071] S1025. Sort the candidate boxes according to the confidence level, and then calculate the intersection-and-union ratio between every two candidate boxes. If the intersection-and-union ratio is greater than a preset intersection-and-union threshold, determine the candidate box with a high confidence level as the target candidate box.

[0072] S1026, calculating the cosine similarity between the clothing feature vector set and the image features in each target candidate frame, and obtaining a captured image of the target person according to the target candidate frame with the highest cosine similarity.

[0073] Exemplarily, the target candidate frame with the highest cosine similarity is used as the positioning frame, and a screenshot of the real-time image is taken according to the positioning frame to obtain a screenshot image of the target person.

[0074] The YOLO model can be used to initially locate the target person in a complex background with relatively few computing resources. At the same time, the data of the YOLO model used for positioning is the same as that of the YOLO model used for appearance extraction, which reduces the difficulty of data processing.

[0075] In some embodiments, the posture recognition model of MMpose includes: a ResNet-50 backbone network and a feature fusion network. The posture analysis of the captured image of the target person is performed through the preset MMpose posture recognition model to generate posture key points and key point coordinates. A posture feature vector set is generated based on the posture key points and key point coordinates. The specific steps include: S1031-S1035.

[0076] S1031 , performing multi-scale pyramid transformation processing on the captured image of the target person to obtain a multi-scale image set.

[0077] S1031. Input the multi-scale image set into the backbone network of the preset MMpose posture recognition model to obtain a multi-layer feature map.

[0078] S1031. Perform adaptive feature fusion processing on the multi-layer feature map to obtain a fused feature map.

[0079] In some embodiments, the MMpose model can also input a clothing feature vector set, perform adaptive feature fusion processing on the multi-layer feature map and the clothing feature vector set, identify the target person's clothing through the clothing feature vector set, reduce the occlusion effect of special fashion styles on human posture, and improve the prediction success rate of posture key points and key point coordinates.

[0080] S1031. Generate a heat map and an offset field according to the fused feature map, decode the heat map and the offset field, and obtain posture key points and key point coordinates.

[0081] S1031. Construct a posture graph structure according to posture key points and key point coordinates, and perform graph convolution processing on the posture graph structure to obtain a posture feature vector set.

[0082] The MMpose model can clearly display the human body posture by annotating 17 key points.

[0083] In some embodiments, the parameter matching model includes: a multi-layer perceptron layer, an attention layer, and a fully connected layer. The clothing feature vector set and the posture feature vector set are input into a preset parameter matching model, and the first shooting parameter is determined according to the clothing feature vector set, and the second shooting parameter is determined according to the posture feature vector set, including: S1041-S1046.

[0084] S1041, performing feature fusion processing on the clothing feature vector set and the posture feature vector set to obtain a fused feature vector set.

[0085] S1041. Input the fused feature vector set into a preset multi-layer perceptron network to obtain an intermediate feature representation.

[0086] S1041. Perform attention network processing on the intermediate feature representation to obtain a weighted feature vector.

[0087] S1041: Input the weighted feature vector into a preset fully connected layer to obtain a predicted value of the shooting parameter.

[0088] S1041 . Perform parameter optimization processing according to the shooting parameter prediction values ​​and preset parameter constraints to obtain optimized shooting parameters.

[0089] S1041: Divide the optimized shooting parameters into first shooting parameters and second shooting parameters.

[0090] Exemplarily, different shooting parameters are determined by clothing features and human posture features, respectively, and exposure parameters, aperture parameters, and sensitivity parameters are determined by clothing features to determine the features of clothing that can be displayed in the captured image. Focal length parameters, shooting angle parameters, and dynamic range parameters are determined by human posture features to reduce the occlusion effect caused by posture and obtain a better photosensitivity effect.

[0091] See also Figure 2 , Figure 2 The embodiment of the present application further provides a schematic block diagram of a shooting and gesture recognition device, wherein the shooting and gesture recognition device 200 is used to execute the above-mentioned shooting and gesture recognition method. The shooting and gesture recognition device 200 can be configured in a server.

[0092] Among them, the server can be an independent server or a server cluster, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms.

[0093] like Figure 2 As shown, the shooting and gesture recognition device 200 includes: a clothing recognition module 201, a person positioning module 202, a gesture recognition module 203, a parameter determination module 204, a time determination module 205 and a shooting control module 206.

[0094] The clothing recognition module 201 is used to input the clothing image into a preset appearance extraction network to perform feature extraction to obtain a clothing feature vector set.

[0095] In some embodiments, the preset appearance extraction network includes: a ResNet network, a conversion network and a feedforward neural network. When the clothing recognition module 201 is used to input the clothing image into the preset appearance extraction network, perform feature extraction, and obtain a clothing feature vector set, it is specifically used to implement: input the clothing image into the ResNet network to obtain a first residual map, a second residual map and a third residual map; perform deformable convolution processing on the third residual map to obtain a deformable fourth residual map; perform spatial attention processing on the deformable fourth residual map to obtain a spatial attention residual map; and perform convolution processing on the spatial attention residual map. The invention discloses a method for obtaining a clothing residual map by performing attention processing to obtain an improved residual map; performing a first convolution processing on the first residual map, the second residual map and the deformable fourth residual map respectively to obtain a first feature set, a second feature set and a third feature set; performing a second convolution processing on the improved residual map to obtain a fourth feature block; combining the first feature block, the second feature block, the third feature block and the fourth feature block to obtain a multi-scale clothing residual map; inputting the multi-scale clothing residual map into the conversion network to obtain enhanced representation information, inputting the enhanced representation information into the feedforward neural network to obtain a clothing feature vector set, the clothing feature vector set including: a category vector and a position prediction vector.

[0096] In some embodiments, when the clothing recognition module 201 is used to implement channel attention processing on the spatial attention residual map to obtain an improved residual map, it is specifically used to implement: average pooling and maximum pooling processing on the spatial attention residual map to obtain a pooling feature set; one-dimensional convolution processing and full connection processing on the pooling feature set to obtain a mapping feature set; element-by-element addition processing on the mapping feature set to obtain a cumulative feature set; sigmoid activation function processing on the cumulative feature set to obtain a one-dimensional channel attention weight block; element-by-element multiplication processing on the spatial attention residual map and the one-dimensional channel attention weight block to obtain an improved residual map.

[0097] In some embodiments, when the clothing recognition module 201 is used to input the enhanced representation information into a feedforward neural network to obtain a clothing feature vector set, and the clothing feature vector set includes: a category vector and a position prediction vector, it is specifically used to implement: processing the enhanced representation information through the fully connected layer of the feedforward neural network to obtain a fully connected output feature; performing an activation function on the fully connected output feature to obtain an activated feature set; performing a classification head processing on the activated feature set to obtain a category vector; performing a regression head processing on the activated feature set to obtain a position prediction vector.

[0098] The person positioning module 202 is used to input the real-time image taken by the camera device into a preset YOLO model, locate the target person in the real-time image, and obtain a captured image of the target person. The preset YOLO model is trained by a clothing feature vector set.

[0099] In some embodiments, the person positioning module 202 is used to input the real-time image taken by the camera device into a preset YOLO model, locate the target person in the real-time image, and obtain a captured image of the target person. When the preset YOLO model is trained by a clothing feature vector set, it is specifically used to implement: extracting features from the real-time image through multiple convolutional layers, and the functions of each convolutional layer include: convolution operation, batch normalization and LeakyReLU activation function to obtain a first feature vector set; down-sampling the first feature vector set through a pooling layer to generate a second feature vector set of multiple scales; for each scale The second feature vector set uses 1x1 convolution to perform channel dimension reduction, and then uses 3x3 convolution to extract features to obtain a third feature vector set; based on the third feature vector set, a 1x1 convolution is used to generate a prediction result, and the prediction result includes: candidate boxes, confidence levels, and category probabilities; the candidate boxes are sorted according to the confidence levels, and then the intersection-and-union ratio between every two candidate boxes is calculated. If the intersection-and-union ratio is greater than a preset intersection-and-union threshold, the candidate box with high confidence is determined as the target candidate box; the cosine similarity between the clothing feature vector set and the image features in each target candidate box is calculated, and a target person capture image is obtained based on the target candidate box with the highest cosine similarity.

[0100] The posture recognition module 203 performs posture analysis on the captured image of the target person through a preset MMpose posture recognition model, generates posture key points and key point coordinates, and generates a posture feature vector set according to the posture key points and key point coordinates.

[0101] In some embodiments, the posture recognition model of MMpose includes: a ResNet-50 backbone network and a feature fusion network. When the posture recognition module 203 is used to implement posture analysis of the captured image of the target person through a preset MMpose posture recognition model, generate posture key points and key point coordinates, and generate a posture feature vector set based on the posture key points and key point coordinates, it is specifically used to implement: performing multi-scale pyramid transformation processing on the captured image of the target person to obtain a multi-scale image set; inputting the multi-scale image set into the backbone network of the preset MMpose posture recognition model to obtain a multi-layer feature map; performing adaptive feature fusion processing on the multi-layer feature map to obtain a fused feature map; generating a heat map and an offset field based on the fused feature map, decoding the heat map and the offset field to obtain posture key points and key point coordinates; constructing a posture graph structure based on the posture key points and key point coordinates, and performing graph convolution processing on the posture graph structure to obtain a posture feature vector set.

[0102] The parameter determination module 204 is used to input the clothing feature vector set and the posture feature vector set into a preset parameter matching model, determine the first shooting parameter according to the clothing feature vector set, and determine the second shooting parameter according to the posture feature vector set.

[0103] In some embodiments, the parameter matching model includes: a multi-layer perceptron layer, an attention layer and a fully connected layer. When the parameter determination module 204 is used to input the clothing feature vector set and the posture feature vector set into the preset parameter matching model, determine the first shooting parameter according to the clothing feature vector set, and determine the second shooting parameter according to the posture feature vector set, it is specifically used to implement: performing feature fusion processing on the clothing feature vector set and the posture feature vector set to obtain a fused feature vector set; inputting the fused feature vector set into a preset multi-layer perceptron network to obtain an intermediate feature representation; performing attention network processing on the intermediate feature representation to obtain a weighted feature vector; inputting the weighted feature vector into a preset fully connected layer to obtain a shooting parameter prediction value; performing parameter optimization processing according to the shooting parameter prediction value and the preset parameter constraint conditions to obtain the optimized shooting parameters; dividing the optimized shooting parameters into a first shooting parameter and a second shooting parameter.

[0104] The time determination module 205 is used to take the time when the similarity between the posture feature vector set and the preset target feature vector set is greater than a preset similarity threshold as the shooting time.

[0105] The shooting control module 206 is used to shoot the target person according to the first shooting parameter and the second shooting parameter at the shooting moment.

[0106] An embodiment of the present application provides an electronic device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement any one of the shooting and gesture recognition methods in the embodiments of the present application when executing the computer program.

[0107] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements any one of the shooting and gesture recognition methods in the embodiments of the present application.

[0108] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A shooting and gesture recognition method, characterized in that: The method comprises: Input the clothing image into the ResNet network of the preset appearance extraction network to obtain a first residual map, a second residual map and a third residual map; perform deformable convolution processing on the third residual map to obtain a deformable fourth residual map; perform spatial attention processing on the deformable fourth residual map to obtain a spatial attention residual map; perform channel attention processing on the spatial attention residual map to obtain an improved residual map; perform first convolution processing on the first residual map, the second residual map and the deformable fourth residual map respectively to obtain a first feature set, a second feature set and a third feature set; perform second convolution processing on the improved residual map to obtain a fourth feature block; combine the first feature block, the second feature block, the third feature block and the fourth feature block to obtain a multi-scale clothing residual map; input the multi-scale clothing residual map into the conversion network of the preset appearance extraction network to obtain enhanced representation information, and input the enhanced representation information into the feedforward neural network of the preset appearance extraction network to obtain a clothing feature vector set, wherein the clothing feature vector set includes: a category vector and a position prediction vector; Inputting the real-time image captured by the camera device into a preset YOLO model, locating the target person in the real-time image, and obtaining a captured image of the target person, wherein the preset YOLO model is a model trained based on the clothing feature vector set; Perform multi-scale pyramid transformation processing on the captured image of the target person to obtain a multi-scale image set; input the multi-scale image set into the ResNet-50 backbone network of the preset MMpose posture recognition model to obtain a multi-layer feature map; perform adaptive feature fusion processing on the multi-layer feature map to obtain a fused feature map; generate a heat map and an offset field according to the fused feature map, decode the heat map and the offset field to obtain posture key points and key point coordinates; construct a posture graph structure according to the posture key points and key point coordinates, perform graph convolution processing on the posture graph structure to obtain a posture feature vector set; Input the clothing feature vector set and the posture feature vector set into a preset parameter matching model, the parameter matching model includes: a multi-layer perceptron layer, an attention layer and a fully connected layer, determine the first shooting parameter according to the clothing feature vector set, and determine the second shooting parameter according to the posture feature vector set, including: performing feature fusion processing on the clothing feature vector set and the posture feature vector set to obtain a fused feature vector set; input the fused feature vector set into the multi-layer perceptron layer to obtain an intermediate feature representation; input the intermediate feature representation into the attention layer for processing to obtain a weighted feature vector; input the weighted feature vector into the fully connected layer to obtain a shooting parameter prediction value; perform parameter optimization processing according to the shooting parameter prediction value and preset parameter constraints to obtain optimized shooting parameters; divide the optimized shooting parameters into the first shooting parameters and the second shooting parameters; The moment when the similarity between the posture feature vector set and the preset target feature vector set is greater than a preset similarity threshold is used as the shooting moment; At the shooting moment, the target person is shot according to the first shooting parameter and the second shooting parameter.

2. The shooting and gesture recognition method according to claim 1, characterized in that: The performing channel attention processing on the spatial attention residual map to obtain an improved residual map includes: Performing average pooling and maximum pooling processing on the spatial attention residual map to obtain a pooled feature set; Performing one-dimensional convolution processing and full connection processing on the pooled feature set to obtain a mapping feature set; Performing element-by-element addition processing on the mapping feature set to obtain an accumulated feature set; Performing sigmoid activation function processing on the accumulated feature set to obtain a one-dimensional channel attention weight block; The spatial attention residual map and the one-dimensional channel attention weight block are subjected to element-by-element multiplication to obtain the improved residual map.

3. The shooting and gesture recognition method according to claim 1, characterized in that: The enhanced representation information is input into the feedforward neural network to obtain the clothing feature vector set, wherein the clothing feature vector set includes: a category vector and a position prediction vector, including: Processing the enhanced representation information through a fully connected layer of the feedforward neural network to obtain a fully connected output feature; Perform activation function processing on the fully connected output features to obtain an activated feature set; Performing classification head processing on the activated feature set to obtain the category vector; The activated feature set is subjected to regression head processing to obtain the position prediction vector.

4. The shooting and gesture recognition method according to claim 1, characterized in that: The real-time image captured by the camera device is input into a preset YOLO model, a target person in the real-time image is located, and a captured image of the target person is obtained, wherein the preset YOLO model is a model trained based on the clothing feature vector set, and includes: Extracting features from the real-time image through multiple convolutional layers, where each convolutional layer performs convolution operation, batch normalization, and LeakyReLU activation function to obtain a first feature vector set; Downsampling the first feature vector set through a pooling layer to generate second feature vector sets of multiple scales; Perform channel dimension reduction on the second feature vector set of each scale using 1x1 convolution, and then extract features using 3x3 convolution to obtain a third feature vector set; Generate a prediction result using 1x1 convolution according to the third feature vector set, wherein the prediction result includes: a candidate box, a confidence level, and a category probability; The candidate frames are sorted according to the confidence, and then the intersection-and-union ratio between every two candidate frames is calculated. If the intersection-and-union ratio is greater than a preset intersection-and-union threshold, the candidate frame with a high confidence is determined as the target candidate frame; The cosine similarity between the clothing feature vector set and the image features in each of the target candidate frames is calculated, and the target person's captured image is obtained according to the target candidate frame with the highest cosine similarity.

5. A shooting and gesture recognition device, characterized in that: Used to implement the shooting and gesture recognition method as described in any one of claims 1 to 4, the shooting and gesture recognition device comprises: A clothing recognition module is used to input clothing images into a preset appearance extraction network to perform feature extraction and obtain a clothing feature vector set; A person positioning module, used for inputting the real-time image captured by the camera device into a preset YOLO model, positioning the target person in the real-time image, and obtaining a captured image of the target person, wherein the preset YOLO model is a model trained based on the clothing feature vector set; A posture recognition module performs posture analysis on the captured image of the target person through a preset MMpose posture recognition model, generates posture key points and key point coordinates, and generates a posture feature vector set according to the posture key points and the key point coordinates; A parameter determination module, used for inputting the clothing feature vector set and the posture feature vector set into a preset parameter matching model, determining a first shooting parameter according to the clothing feature vector set, and determining a second shooting parameter according to the posture feature vector set; A moment determination module, used for taking the moment when the similarity between the posture feature vector set and the preset target feature vector set is greater than a preset similarity threshold as the shooting moment; A shooting control module is used to shoot the target person according to the first shooting parameter and the second shooting parameter at the shooting moment.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the shooting and gesture recognition method as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the shooting and gesture recognition method according to any one of claims 1 to 4.

Images