Outdoor shooting assistance system and method based on image recognition

Abstract

The application provides an outdoor shooting auxiliary system and method based on image recognition, respectively extracts the spatial distribution features of shooting posture nodes in visible light images and infrared thermal images, and then determines the distribution difference loss of the shooting posture nodes between different heterogeneous images; when the image adopts process is interfered by strong light, the element labels of the shooting posture nodes in different heterogeneous images are fuzzy registered based on all the distribution difference losses, the attribution confidence of the shooting posture nodes is obtained, and then the pixel constraint condition of the shooting posture nodes in different heterogeneous images when the interference is suppressed is determined according to all the attribution confidences; the spatial distribution features of the shooting posture nodes in different heterogeneous images are hierarchically constrained and fused based on the cross-modal fusion attention between different heterogeneous images and the pixel constraint condition of the shooting posture nodes, and the fusion features of the shooting posture are obtained. By adopting the scheme of the application, hierarchical constraint fusion of the shooting posture node features between different heterogeneous images can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, more particularly, the present application relates to an outdoor shooting auxiliary system and method based on image recognition. BACKGROUND

[0002] In the outdoor shooting auxiliary system, accurate recognition of shooting posture is of great significance to improve training quality and combat efficiency. As a key means to realize posture recognition, image recognition can quickly and non-contactly extract key action nodes by collecting posture image information of the shooter, providing real-time and accurate posture feedback for the system. In the complex outdoor environment, image recognition can not only improve the intelligent degree of the training process, but also assist the instructor or the system to automatically evaluate the standardization and stability of shooting actions, which is one of the important directions of intelligent shooting auxiliary system construction.

[0003] In the prior art, there are still significant technical obstacles in applying image recognition methods to recognize shooting posture in outdoor environments. Strong sunlight, shadow changes and fluctuations in weather conditions can seriously interfere with the quality of visible light images, making key posture node information blurred or missing, which is difficult to support high-precision recognition. Although infrared thermal images can effectively compensate for these problems, cross-modal fusion and registration between different types of images is still a major problem. The existing technology usually relies on traditional registration methods, but these methods are difficult to accurately cope with the distribution differences between images in complex environmental conditions, especially in strong light interference, resulting in poor registration results, which affects the accuracy of the final posture recognition. Therefore, how to realize the hierarchical constraint fusion of shooting posture node features between different heterogeneous images has become a problem in the industry. SUMMARY

[0004] The present application provides an outdoor shooting auxiliary system and method based on image recognition, which can realize hierarchical constraint fusion of shooting posture node features between different heterogeneous images.

[0005] In a first aspect, the present application provides an image recognition method for shooting posture, which is used for image recognition of shooting posture of a shooter in an outdoor shooting auxiliary system, comprising the following steps:

[0006] Synchronously collecting a visible light image and an infrared thermal image of the shooting posture of the shooter in an outdoor shooting environment;

[0007] Respectively performing multi-scale convolution on the visible light image and the infrared thermal image to extract the spatial distribution features of the shooting posture nodes in the visible light image and the infrared thermal image, and then determining the distribution difference loss of the same shooting posture node between different heterogeneous images;

[0008] In the image adoption process, the element labels of the shooting posture nodes in different heterogeneous images are fuzzy matched based on all the distribution difference losses when the images are interfered by strong light, to obtain the attribution confidence of the shooting posture nodes in different heterogeneous images, and then the pixel constraint conditions of the shooting posture nodes in different heterogeneous images in interference suppression are determined according to all the attribution confidences;

[0009] The cross-modal fusion attention of the shooting posture nodes between different heterogeneous images is generated based on the self-attention mechanism, the spatial distribution features of the shooting posture nodes in different heterogeneous images are hierarchically constrained and fused based on the cross-modal fusion attention and the pixel constraint conditions of the shooting posture nodes, and the fusion features of the shooting posture are obtained.

[0010] Preferably, the spatial distribution features of the shooting posture nodes in the visible light image and the infrared thermal image are extracted, and then the distribution difference loss of the same shooting posture node between different heterogeneous images specifically includes:

[0011] A convolutional neural network module with multi-scale receptive field is constructed;

[0012] The spatial distribution features of the shooting posture nodes in the visible light image and the infrared thermal image are extracted by the convolutional neural network module;

[0013] The node label vector of the shooting posture node in the visible light image is determined according to the spatial distribution features corresponding to the visible light image;

[0014] The node label vector of the shooting posture node in the infrared thermal image is determined according to the spatial distribution features corresponding to the infrared thermal image;

[0015] The distribution difference loss of the same shooting posture node between different heterogeneous images is determined by all the node label vectors.

[0016] Preferably, the element labels of the shooting posture nodes in different heterogeneous images are fuzzy matched based on all the distribution difference losses, to obtain the attribution confidence of the shooting posture nodes in different heterogeneous images, and the pixel constraint conditions of the shooting posture nodes in different heterogeneous images in interference suppression are determined according to all the attribution confidences.

[0017] The fuzzy membership of the shooting posture nodes between different heterogeneous images is determined from all the distribution difference losses;

[0018] The element labels of the shooting posture nodes in different heterogeneous images are fuzzy matched and inferred according to all the fuzzy memberships, to obtain the label confidence between the shooting posture nodes;

[0019] The attribution confidence of the shooting posture nodes in different heterogeneous images is determined by all the label confidences.

[0020] Preferably, the pixel constraint conditions of the shooting posture node of different heterogeneous images in interference suppression according to all the attribution confidence degrees specifically include:

[0021] The pixel constraint conditions of the shooting posture node of different heterogeneous images in interference suppression according to all the attribution confidence degrees are determined.

[0022] The pixel constraint conditions of the shooting posture node of different heterogeneous images in interference suppression according to all the attribution confidence degrees are determined.

[0023] The pixel constraint conditions of the shooting posture node of different heterogeneous images in interference suppression according to all the attribution confidence degrees are determined.

[0024] Preferably, the cross-modal fusion attention of the shooting posture node between different heterogeneous images based on the self-attention mechanism specifically includes:

[0025] The cross-modal fusion attention of the shooting posture node between different heterogeneous images based on the self-attention mechanism specifically includes:

[0026] The cross-modal fusion attention of the shooting posture node between different heterogeneous images based on the self-attention mechanism specifically includes:

[0027] Preferably, the visible light image is collected by a camera.

[0028] Preferably, the infrared thermal image is collected by an infrared thermal imager.

[0029] In a second aspect, the present application provides an outdoor shooting auxiliary system based on image recognition, which comprises an image recognition unit, and the image recognition unit comprises:

[0030] A collection module is configured to synchronously collect a visible light image and an infrared thermal image of a shooting posture of a shooter in an outdoor shooting environment.

[0031] A processing module is configured to perform multi-scale convolution on the visible light image and the infrared thermal image respectively, extract spatial distribution features of a shooting posture node in the visible light image and the infrared thermal image, and then determine a distribution difference loss of a same shooting posture node between different heterogeneous images.

[0032] The processing module is further configured to, when the image is interfered by strong light during image adoption, perform fuzzy registration on element labels of the shooting posture node in different heterogeneous images based on all distribution difference losses, to obtain attribution confidence of the shooting posture node in different heterogeneous images, and then determine pixel constraint conditions of the shooting posture node in different heterogeneous images during interference suppression according to all attribution confidences.

[0033] The execution module is configured to generate cross-modal fusion attention of the shooting posture node between different heterogeneous images based on the self-attention mechanism, perform hierarchical constraint fusion on spatial distribution features of the shooting posture node in different heterogeneous images based on the cross-modal fusion attention and the pixel constraint conditions of the shooting posture node, and obtain fusion features of the shooting posture.

[0034] In a third aspect, a computer device is provided, which includes a memory and a processor, the memory stores a code, and the processor is configured to acquire the code and execute the image recognition method of the shooting posture.

[0035] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the image recognition method of the shooting posture.

[0036] The technical scheme provided by the embodiments of the present application has the following beneficial effects:

[0037] In the embodiments of the present application, first, visible light images and infrared thermal images of shooting postures of a shooter in an outdoor shooting environment are synchronously collected; multi-scale convolution is respectively performed on the visible light images and the infrared thermal images to extract spatial distribution features of shooting posture nodes in the visible light images and the infrared thermal images, and then distribution difference losses of a same shooting posture node between different heterogeneous images are determined; when the images are interfered by strong light during image adoption, element labels of the shooting posture node in different heterogeneous images are registered based on all distribution difference losses to obtain attribution confidence of the shooting posture node in different heterogeneous images, and then pixel constraint conditions of the shooting posture node in different heterogeneous images during interference suppression are determined according to all attribution confidences; cross-modal fusion attention of the shooting posture node between different heterogeneous images is generated based on the self-attention mechanism, and spatial distribution features of the shooting posture node in different heterogeneous images are hierarchically and constraintly fused based on the cross-modal fusion attention and the pixel constraint conditions of the shooting posture node to obtain fusion features of the shooting posture.

[0038] Therefore, this application achieves hierarchical constraint fusion of the spatial distribution features of shooting posture nodes in different heterogeneous images by fusing attention across modalities and pixel constraints of shooting posture nodes, thus obtaining the fused features of shooting posture. First, by simultaneously acquiring visible light images and infrared thermal images, the system can supplement the image information lost due to illumination variations in outdoor environments with drastic changes in lighting, thereby establishing a redundant but complementary multimodal image foundation. Second, the spatial distribution features of shooting posture nodes in visible light images and infrared thermal images are extracted separately, thereby determining the spatial distribution features of the same shooting posture node in different... The system employs a distribution difference loss mechanism among heterogeneous images. By performing multi-scale convolution on images of different modalities, it can acquire the spatial distribution features of pose nodes at different scales and further analyze the distribution difference loss across different modalities, thereby achieving explicit characterization of node features across heterogeneous images. Then, based on the distribution difference loss of shooting pose nodes across different heterogeneous images, fuzzy registration is performed on the element labels of shooting pose nodes in different heterogeneous images to obtain the attribution confidence of shooting pose nodes in different heterogeneous images. Finally, based on all the attribution confidence scores, the image of shooting pose nodes in different heterogeneous images under interference suppression is determined. Under strong light interference, a fuzzy registration mechanism is introduced to construct the attribution confidence of the shooting posture nodes based on the distribution differences of the shooting posture nodes in different heterogeneous images. This avoids misjudgment of key nodes due to direct registration failure, thus forming a pixel-level constraint based on confidence, which makes the fusion more fault-tolerant and stable. Finally, the spatial distribution features of the shooting posture nodes in different heterogeneous images are hierarchically constrained and fused through cross-modal fusion attention and pixel constraints of the shooting posture nodes to obtain the fused features of the shooting posture. A self-attention mechanism is then used to generate cross-modal fusion attention. While focusing on the semantic correlation between different heterogeneous images, and combining the aforementioned pixel constraints, the pose node features in different images are constrained hierarchically to achieve a fusion process from coarse to fine and from low to high, gradually improving the semantic consistency and spatial alignment accuracy of the fusion results. Furthermore, the fused shooting pose features, while maintaining the diversity of the original images, significantly enhance the robustness and accuracy of the system for pose recognition in complex environments, achieving the technical objective of hierarchical constraint fusion of shooting pose nodes between heterogeneous images. In summary, the scheme of this application can realize hierarchical constraint fusion of shooting pose node features between different heterogeneous images. Attached Figure Description

[0039] Figure 1 This is an exemplary flowchart of an image recognition method for shooting postures according to some embodiments of this application;

[0040] Figure 2 This is a flowchart illustrating the working application of an outdoor shooting assistance system according to some embodiments of this application;

[0041] Figure 3 is a flowchart of determining the home confidence according to some embodiments of the present application;

[0042] Figure 4 is a structural diagram of an image recognition unit according to some embodiments of the present application;

[0043] Figure 5 is a structural diagram of a computer device for implementing the image recognition method of the shooting posture according to some embodiments of the present application. DETAILED DESCRIPTION

[0044] In order to better understand the technical solutions of the present application, the technical solutions of the present application will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0045] Reference Figure 1 The figure is an exemplary flowchart of the image recognition method of the shooting posture 100 according to some embodiments of the present application, which mainly includes the following steps:

[0046] In step 101, visible light images and infrared thermal images of the shooting posture of the shooter in an outdoor shooting environment are synchronously collected.

[0047] It should be noted that the synchronous collection in the present application refers to the collection of different types of image data (such as visible light images and infrared thermal images) at the same time through the use of a collection device with time synchronization function, to ensure the accurate alignment of the two images in time, through synchronous collection, the image data obtained at the same time node can have consistent time stamps, so that the images of different modalities can be correctly matched in subsequent processing and analysis, avoiding the problem of inconsistent image features due to time difference in collection; it should also be noted that the visible light images in the present application are collected by a camera, which can be a visible light camera, which is not limited here, and the infrared thermal images in the present application are collected by an infrared thermal imager.

[0048] In specific implementation, the synchronous collection of the visible light images and the infrared thermal images of the shooting posture of the shooter in the outdoor shooting environment can be implemented in the following manner, that is, a dual-spectrum imager with synchronous collection function can be configured to achieve the synchronous collection, the dual-spectrum imager integrates a visible light camera and an infrared thermal imager, the visible light camera is used to collect the visible light images of the shooting posture, and the infrared thermal imager is used to collect the infrared thermal images of the shooting posture, the visible light camera and the infrared thermal imager can be started at the same time through a time synchronization control program in the dual-spectrum imager, and the image collection is automatically started when the shooter starts the gun action, and the synchronous collection is completed.

[0049] In some embodiments, referring to Figure 2 As shown in the figure, which is a working application flowchart of an outdoor shooting auxiliary system in some embodiments of the present application, the figure shows the working application flow of the outdoor shooting auxiliary system. In an outdoor shooting environment 110, a shooter performs a shooting action, and the posture and other related information of the shooter are collected by a visible light camera 120 and an infrared thermal imager 130 respectively. The former is used to obtain a visible light image, and the latter is used to obtain an infrared thermal image, so as to cope with complex lighting conditions. Then, the collected two types of image data are transmitted to a data processor 140. The processor realizes fusion processing of different modal images, so as to extract accurate shooting posture features. Finally, a visualization module 150 visually displays the processed posture evaluation results. Through the fusion of visible light and infrared thermal image information, the entire working process of the system is completed, thereby providing auxiliary support for outdoor shooting.

[0050] In step 102, multi-scale convolution is performed on the visible light image and the infrared thermal image respectively, and the spatial distribution features of the shooting posture nodes in the visible light image and the infrared thermal image are extracted, and then the distribution difference loss of the same shooting posture node between different heterogeneous images is determined.

[0051] It should be noted that the multi-scale convolution of the visible light image and the infrared thermal image in the present application refers to using convolution kernels of different sizes to perform convolution processing on the visible light image and the infrared thermal image respectively, and extracting feature information of the respective images under different scale convolutions.

[0052] In some embodiments, the extraction of the spatial distribution features of the shooting posture nodes in the visible light image and the infrared thermal image, and the determination of the distribution difference loss of the same shooting posture node between different heterogeneous images can be realized by the following steps:

[0053] A convolutional neural network module with multi-scale receptive field is constructed;

[0054] The spatial distribution features of the shooting posture nodes in the visible light image and the infrared thermal image are extracted by the convolutional neural network module;

[0055] The node label vector of the shooting posture node in the visible light image is determined according to the spatial distribution features corresponding to the visible light image;

[0056] The node label vector of the shooting posture node in the infrared thermal image is determined according to the spatial distribution features corresponding to the infrared thermal image;

[0057] The distribution difference loss of the same shooting posture node between different heterogeneous images is determined by all node label vectors.

[0058] It should be noted that the shooting posture node in the present application refers to a feature point for representing the position of a key part of a shooter's body (such as the shoulder, elbow, knee, wrist) in the image space; the spatial distribution feature in the present application refers to a feature for measuring the position structure layout of the shooting posture node in the image; the node label vector in the present application refers to a vector for representing the position distribution feature of the shooting posture node in the image space; and the distribution difference loss in the present application refers to a quantitative index for measuring the difference in the spatial position of the same shooting posture node between different heterogeneous images.

[0059] In a specific implementation, first, the convolutional neural network module with a multi-scale receptive field can be implemented in the following manner: an HRNet architecture with multi-scale feature extraction capability can be selected, and in the present embodiment, first, a backbone network maintaining a high-resolution feature flow is built, and a standard convolution operation is performed on the input image in the initial stage of the network to extract basic features; second, a plurality of parallel branches (the convolution kernel sizes of the branches are different) are designed, each branch processes a feature map of a different resolution, multi-scale modeling is achieved by increasing the stride or downsampling layer by layer, and the spatial structure expression capability is enhanced by information exchange between the branches, i.e., a multi-resolution fusion mechanism, and the features of different scales are complemented to each other; finally, the fused multi-scale features are sent to a high-resolution output layer to generate a distribution feature map of the shooting posture node with rich spatial perception capability; the spatial distribution features of the shooting posture node in the visible light image and the infrared thermal image can be extracted by the convolutional neural network module in the following manner: the visible light image and the infrared thermal image are input into the convolutional neural network module as input parameters, the distribution information of the shooting posture node in the visible light image and the infrared thermal image is output by the output layer of the convolutional neural network module, and the distribution information is taken as the spatial distribution features of the shooting posture node in the corresponding image; the node label vector of the shooting posture node in the visible light image can be determined according to the spatial distribution features corresponding to the visible light image in the following manner: for the visible light image, the node labels of the shooting posture node in the spatial distribution features are arranged into a vector according to the distribution positions of the pre-labeled shooting posture nodes, and the vector is taken as the node label vector of the shooting posture node in the visible light image; the node label vector of the shooting posture node in the infrared thermal image can be determined according to the spatial distribution features corresponding to the infrared thermal image in the following manner: for the infrared thermal image, the node labels of the shooting posture node in the spatial distribution features are arranged into a vector according to the distribution positions of the pre-labeled shooting posture nodes, and the vector is taken as the node label vector of the shooting posture node in the infrared thermal image.

[0060] Preferably, in the above embodiment, the distribution difference loss of the same shooting posture node between different heterogeneous images can be determined by all the node label vectors in the following steps:

[0061] determine the distribution difference entropy of the shooting posture nodes between different heterogeneous images by all node label vectors;

[0062] extract the distribution difference amount of the shooting posture nodes in different heterogeneous images for each shooting posture node;

[0063] determine the distribution difference loss of the shooting posture nodes between different heterogeneous images by the distribution difference amount and the distribution difference entropy, and then obtain the distribution difference loss of each shooting posture node between different heterogeneous images.

[0064] It should be noted that the distribution difference entropy in the present application is an index for measuring the uncertainty of the position distribution of the shooting posture nodes between different heterogeneous images.

[0065] In a specific implementation, the distribution difference entropy of the shooting posture nodes between different heterogeneous images determined by all node label vectors can be implemented in the following manner, that is, the Euclidean distance can be used as the difference value between each two corresponding elements of two node label vectors, and all the difference values are substituted into the information entropy calculation formula, and the calculated information entropy is taken as the distribution difference entropy of the shooting posture nodes between different heterogeneous images. The extraction of the distribution difference amount of the shooting posture nodes in different heterogeneous images can be implemented in the following manner, that is, the Euclidean distance between the distribution positions of the shooting posture nodes in different heterogeneous images can be taken as the distribution difference amount of the position. The determination of the distribution difference loss of the shooting posture nodes between different heterogeneous images by the distribution difference amount and the distribution difference entropy can be implemented in the following manner, that is, for each shooting posture node, the distribution difference amount of the node in different images and the distribution difference entropy of all shooting posture nodes can be calculated by weighting, and the weighted result is taken as the distribution difference loss of the node between different heterogeneous images, wherein the weighting weight can be set in advance according to historical operation data, which is not described here. Through the above manner, the distribution difference loss of each shooting posture node between different heterogeneous images can be obtained.

[0066] It should be noted that the present application embodiment introduces the distribution modeling of the node label vector, not only quantifies the spatial position difference of the shooting posture nodes between different modal images, but also introduces the node overall distribution stability evaluation by fusing the distribution difference entropy. Compared with the existing technology based on position or feature similarity alignment, the uncertainty and offset trend of the nodes under cross-modal images can be more comprehensively reflected, so that more robust and higher precision shooting posture recognition can be realized in a strong interference environment.

[0067] In step 103, when the image is disturbed by strong light in the image acquisition process, the element labels of the shooting posture nodes in different heterogeneous images are fuzzy registered based on all the distribution difference losses to obtain the attribution confidence of the shooting posture nodes in different heterogeneous images, and then the pixel constraint condition of the shooting posture nodes in different heterogeneous images in interference suppression is determined according to all the attribution confidences.

[0068] It should be noted that the strong light interference in the present application refers to the phenomenon of overexposure, light spot or detail loss in the image under the action of a high-intensity light source. In an outdoor shooting environment, due to direct sunlight, object reflection or other high-intensity light source irradiation, the collected image appears overexposure, uneven brightness or detail loss in the local area, thereby affecting the recognition accuracy of the shooting posture node features in the image.

[0069] In some embodiments, as shown in the figure, which is a flowchart for determining the attribution confidence in some embodiments of the present application, the element labels of the shooting posture nodes in different heterogeneous images are fuzzy registered based on all the distribution difference losses to obtain the attribution confidence of the shooting posture nodes in different heterogeneous images, which can be realized by the following steps: Figure 3 In step 1031, the fuzzy membership of the shooting posture nodes between different heterogeneous images is determined from all the distribution difference losses.

[0070] In step 1032, the element labels of the shooting posture nodes in different heterogeneous images are fuzzy registered and inferred according to all the fuzzy memberships to obtain the label confidence between the shooting posture nodes.

[0071] In step 1033, the attribution confidence of the shooting posture nodes in different heterogeneous images is determined by all the label confidences.

[0072] It should be noted that the fuzzy membership in the present application refers to the attribution possibility value of the shooting posture nodes in different modal images relative to the real node distribution; the element label in the present application refers to the annotation information for describing the spatial position, category and feature attribute of the shooting posture nodes in the image; and the label confidence in the present application refers to an index for measuring the matching reliability degree of the labels of the shooting posture nodes in different modal images.

[0073]

[0074] ​In a specific implementation, first, the fuzzy membership of the shooting posture nodes between different heterogeneous images determined by all distribution difference losses can be implemented in the following manner, that is, the distribution difference loss of each shooting posture node between different heterogeneous images calculated in the foregoing can be used as an input variable of the fuzzy system. In order to convert the distribution difference loss into a measurable uncertainty membership, a fuzzy membership function library is constructed, and common function forms include a Gaussian function and an S-type membership function. The smaller the difference is, the greater the membership value corresponding to the difference is. In the embodiments of the present application, the fuzzy membership function is a Gaussian function, and in other embodiments, other fuzzy membership functions can also be used, which are not limited here. For the Gaussian function, the center is set to zero difference, and the standard deviation is set according to the training image statistics, so that the membership can be mapped to the possibility that the shooting posture node belongs to the target node in different heterogeneous image modalities (i.e., visible light images and infrared thermal images). This process makes the registration between nodes no longer a hard decision of “whether to match”, but a fuzzy judgment of “matching degree”. Second, the element label of the shooting posture node in different heterogeneous images is inferred according to all fuzzy memberships to obtain the label confidence between shooting posture nodes, which can be implemented in the following manner, that is, after the fuzzy membership of the shooting posture node in all modal images is calculated, the different fuzzy memberships are input into the fuzzy inference system (FIS) in the prior art for fusion decision-making. In this process, the position coordinates, confidence and edge gradient label attributes of the shooting posture node in each modal image are used as input elements (i.e., element labels), and the corresponding fuzzy membership is used for inference by using a rule library, for example, “if the A image membership is high and the B image difference loss is large, then the A image label is preferentially used”. After inference, a continuous value label confidence is output, which is used to represent the label matching confidence level of the shooting posture node between different image modalities, avoiding matching failure caused by distortion of a single label feature. Then, the attribution confidence of the shooting posture node in different heterogeneous images is determined by all label confidences, which can be implemented in the following manner, that is, the label confidences of the shooting posture node in all modal images are used as evaluation indexes, and the maximum confidence attribution method is used to finally attribute and infer the label of each node to obtain the attribution confidence of the shooting posture node in different heterogeneous images. For example, after the label confidences of each shooting posture node in each image modality are normalized, the one with the maximum confidence is selected as the node attribution, or a fused label is synthesized in proportion as the final representation form of the node. This process realizes image node alignment under fuzzy conditions and effectively alleviates the label mismatching problem caused by modal interference (such as overexposure of a visible light image and blurring of an infrared thermal image).

[0075] It should be noted that by introducing the fuzzy membership degree and label credibility reasoning mechanism based on distribution difference loss in the strong light interference scene, the node information loss problem caused by overexposure, reflection and the like of the visible light image can be effectively alleviated, the stable node distribution characteristics in the infrared thermal image are fused, and the flexible alignment of the node element label is realized in combination with the fuzzy registration, so that the system can still determine the credible source of the node information in each modality image according to the fuzzy membership degree when facing the serious distortion of the local image, and then accurately generate the attribution confidence. Compared with the traditional technology relying on a single modality or rigid feature matching, the shooting posture recognition stability is significantly improved under strong light interference.

[0076] In some embodiments, the pixel constraint condition of the shooting posture node of different heterogeneous images in interference suppression according to all attribution confidences can be implemented by the following steps:

[0077] According to all attribution confidences, the interference suppression parameters of the shooting posture node pixels in different heterogeneous images are subjected to multi-objective collaborative constraint modeling, and a pixel constraint condition set is obtained;

[0078] Based on the multi-objective optimization algorithm, the pixel constraint condition set is subjected to interference suppression parameter optimization, and the optimal suppression parameter of the shooting posture node pixels in different heterogeneous images is obtained;

[0079] The pixel constraint condition of the shooting posture node of different heterogeneous images in interference suppression is determined by all optimal pixel suppression parameters.

[0080] It should be noted that the interference suppression parameter in the present application refers to a control factor for adjusting the pixel characteristics of the shooting posture node region in the image to reduce the influence of strong light interference; the optimal suppression parameter in the present application refers to the best parameter combination obtained through the optimization process, which is used to reduce the interference in the image and enhance the feature information of the shooting posture node; the pixel constraint condition in the present application refers to the adjustment limit set for the image pixel region in the image processing process to ensure the accuracy of the feature of the shooting posture node under interference suppression.

[0081] In a specific implementation, first, the interference suppression parameters of the pixels of the shooting posture nodes in different heterogeneous images are modeled according to all the attribution confidences, a multi-objective collaborative constraint model is established, and a pixel constraint condition set is obtained. The pixel constraint condition set can be obtained in the following manner: the attribution confidence is used to evaluate the reliability of each shooting posture node in the visible light image and the infrared thermal image. The higher the attribution confidence, the more reliable the information of the node in the image modal. The pixel feature parameters of the region corresponding to each posture node are extracted from each image, such as the luminance value (L), the gradient amplitude (G), the local contrast (C), and the infrared thermal response intensity (T). The attribution confidence is normalized, and the normalized value is used as the weight coefficient of the pixel feature parameters of the region corresponding to the posture node. The weighted pixel feature parameters are combined to form an interference-sensitive feature vector. A multi-objective collaborative constraint model is established through all the interference-sensitive feature vectors. The goal is to maximize the effective node pixel information and minimize the false response or distortion area caused by strong light interference. The modeling process can be realized by using the normalized linear weighting or the Pareto frontier construction method. The condition set obtained by modeling is used as the pixel constraint condition set. Then, the pixel constraint condition set is optimized based on a multi-objective optimization algorithm to obtain the optimal suppression parameters of the pixels of the shooting posture nodes in different heterogeneous images. The optimization can be realized in the following manner: the constraint condition set is used as the optimization input, a multi-objective evolutionary optimization algorithm in the prior art, such as NSGA-II (non-dominated sorting genetic algorithm), is introduced, the pixel features of each shooting posture node in different modal images are used as decision variables, and the optimization goals are set: minimizing the luminance saturation, maximizing the thermal image edge definition, and maximizing the SNR (signal-to-noise ratio) of the posture region. In the iteration process, the optimal pixel regulation parameter combination that meets the constraint condition is found through the mutation, crossover, and selection mechanisms. The optimal pixel regulation parameter combination is used as the optimal suppression parameter of the pixels of the shooting posture nodes. The optimization result is the optimal interference suppression strategy of each node in each image modal, such as the exposure compensation value, the filtering threshold, and the image enhancement gain, so as to accurately control the visual response of the local image region of the node. Finally, the pixel constraint conditions of the shooting posture nodes in different heterogeneous images during interference suppression are determined through all the optimal pixel suppression parameters. The optimal pixel suppression parameters output by the optimization are mapped to the corresponding shooting posture nodes in the images, and the pixel constraint conditions of the shooting posture nodes in different heterogeneous images are established through a unified pixel regulation strategy.It should be noted that the pixel adjustment strategy in the present application refers to a technical method for suppressing interference and enhancing features by controlling image pixel attributes when the shooting posture node region is disturbed in the image (such as strong light interference), the core of which is to use optimal pixel suppression parameters to adaptively adjust the brightness, contrast and texture definition of the node region. In terms of technical implementation, the pixel adjustment strategy combines thermal feature enhancement in the infrared thermal image with brightness suppression in the visible light image based on the local region attribution confidence, adjusts the pixels of the shooting posture node region by using local histogram equalization, and completes the stability and distinguishability of the enhanced posture feature in the interference environment. Through the pixel adjustment strategy, consistent visual expression of the shooting posture node region can be formed in the multi-modal image, so that the subsequent fusion recognition link has stronger robustness and positioning accuracy.

[0082] It should be noted that the scheme of the present application accurately formulates pixel constraint conditions for interference suppression for shooting posture nodes in different heterogeneous images by combining attribution confidence and multi-objective optimization algorithm, which has the following significant advantages compared with the prior art: first, the reliability of the node feature is quantified by the attribution confidence, avoiding information loss or excessive suppression that may be caused by global adjustment in traditional methods; second, the multi-objective collaborative constraint modeling and optimization algorithm can accurately balance interference suppression and feature preservation, and individualize processing for local features of each shooting posture node, thereby improving the stability and accuracy of posture recognition; finally, the optimized pixel suppression parameters can adaptively cope with strong light interference, avoiding insufficient or excessive suppression effect in traditional single mode, and enhancing the robustness in complex environments.

[0083] In step 104, cross-modal fusion attention of shooting posture nodes between different heterogeneous images is generated based on a self-attention mechanism, and spatial distribution features of shooting posture nodes in different heterogeneous images are hierarchically constrained and fused based on the cross-modal fusion attention and pixel constraint conditions of shooting posture nodes, to obtain fusion features of shooting postures.

[0084] In some embodiments, the cross-modal fusion attention of shooting posture nodes between different heterogeneous images generated based on the self-attention mechanism can be implemented by the following steps:

[0085] The attention weight matrix of the shooting posture node in different heterogeneous images is generated based on the self-attention mechanism;

[0086] The feature attention of shooting posture nodes between different heterogeneous images is fused by all attention weight matrices to obtain cross-modal fusion attention of shooting posture nodes between different heterogeneous images.

[0087] It should be noted that the cross-modal fusion attention in the present application is an importance weight index for measuring the feature representation significance of the same shooting posture node in different modal images.

[0088] In a specific implementation, first, the attention weight matrix of the shooting posture node in different heterogeneous images generated based on the self-attention mechanism can be implemented in the following manner, that is, the visible light image and the infrared thermal image are input into a pre-trained feature extraction network (such as HRNet) respectively to obtain a high-dimensional representation tensor F vis of the shooting posture node ir , wherein each node feature constitutes a vector, and for the node feature of each image modality, a Query (Q) matrix, a Key (K) matrix and a Value (V) matrix are constructed through linear transformation: Q = W q F, K = W k F, and V = W v F, and the standard self-attention mechanism formula is used to calculate the attention weight matrix A vis and A ir within the modal, wherein d is the dimension size of the Key vector, and A vis and A ir respectively represent the relative importance of the node features within the image modality, the attention weight matrices of the two modalities are used for weighting the value vector of the node respectively to obtain the feature attention representation of each modality: F′ vis = A vis · V vis , F′ ir = A ir · V ir ; then, the feature attention of the shooting posture node between different heterogeneous images is fused through all the attention weight matrices to obtain the cross-modal fusion attention of the shooting posture node between different heterogeneous images, which can be obtained by fusing (such as weighted average, in the present embodiment, the weight is set to the same value) F′ vis and F′ ir to obtain the representation F fusion of the cross-modal fusion attention, which simultaneously considers the importance and context correlation of the node under the two modalities, and enhances the robust expression ability of the model to the shooting posture node feature under a complex interference scene, it should be further explained that W q , W k , and W v respectively represent the learnable weight matrices for mapping the input features to the Query (query), Key (key) and Value (value) spaces, which are essentially linear transformation parameters for generating attention weights in the self-attention mechanism, and their functions are as follows: W q converts the node feature into a Query expression, which represents the content to be “focused” by the current node; W kconvert the node features into Key expression, representing the "information provided by the current node"; W v convert the node features into Value expression, which is the information finally weighted and fused.

[0089] In some embodiments, the pixel constraint condition of the cross-modal fusion attention and shooting posture node is used to hierarchically constrain and fuse the spatial distribution features of the shooting posture nodes in different heterogeneous images, and the fusion features of the shooting posture can be obtained by using the following steps:

[0090] The cross-modal fusion attention of the shooting posture nodes between different heterogeneous images is suppressed and compensated by the pixel constraint condition of the shooting posture node, and the compensated cross-modal attention weight is obtained.

[0091] The spatial distribution features of the shooting posture nodes in different heterogeneous images are hierarchically semantically aligned to generate structural feature maps under each modality.

[0092] The structural feature maps under each modality are weighted and fused according to the compensated attention weight, and the fusion features of the shooting posture are obtained.

[0093] It should be noted that the suppression and compensation in the present application means reducing the response attention of the interference area and enhancing the response attention of the effective area; the hierarchical semantic alignment in the present application means that the node semantic information corresponding to different modal images at multiple feature levels is structure matched.

[0094] In a specific implementation, first, the cross-modal fusion attention of the shooting posture node between different heterogeneous images is inhibited and compensated by the pixel constraint condition of the shooting posture node, and the compensated cross-modal attention weight can be realized in the following manner, that is, the cross-modal fusion attention is inhibited and compensated by the pixel constraint condition of the shooting posture node, and technically, the pixel mask weighting is used to inhibit the redundant attention region, and a higher weight is given to the effective region to generate the compensated cross-modal attention weight. It needs to be further explained that the pixel mask weighting inhibition refers to introducing a pixel mask generated by the pixel constraint condition of the shooting posture node on the basis of the cross-modal attention distribution, and performing a structural intervention on the attention distribution through an element-by-element weighting operation. Specifically, a corresponding binary or continuous value mask is constructed according to the optimal pixel inhibition parameter determined in each heterogeneous image, and the mask value represents the confidence of the current pixel in the target posture region. Then, the mask is multiplied with the original cross-modal attention matrix pixel by pixel to realize the explicit inhibition of the attention of the redundant or irrelevant region, and to enhance the response strength of the key region. It needs to be further explained that the method of the application not only retains the dynamic weight distribution advantage of the attention mechanism, but also introduces spatial constraints to ensure that the attention is concentrated in the high-confidence shooting node region, thereby improving the accuracy and stability of the cross-modal feature fusion. Second, the spatial distribution features of the shooting posture node in different heterogeneous images are aligned in layers, and the structural feature maps under each modality can be realized in the following manner, that is, considering the natural differences in geometric distribution and perception scale between visible light images and infrared thermal images, in order to realize their consistent expression in a unified feature space, the embodiment adopts a multi-scale feature extraction network (such as U-Net) to extract the shallow geometric structure information and deep semantic features of each modality image, and introduces a differentiable spatial transformation module (such as Spatial Transformer Network, STN) to perform affine alignment operations such as rotation, translation and scaling on each modality feature map, thereby realizing the consistency of multi-modal features in spatial structure. After the above alignment processing, the structural features of each modality are uniformly represented as a structural feature map, that is, the shooting posture features represented in different image modalities are clear in scale level and spatial distribution alignment. Then, the structural feature maps under each modality are weighted and fused according to the compensated attention weight to obtain the fusion features of the shooting posture, which can be realized in the following manner, that is, the structural feature maps of each modality can be weighted and summed channel by channel by using an adaptive fusion module (such as Attention-Guided Fusion), and a residual connection is introduced in the fusion process to retain the original feature structure and avoid the disappearance of key semantic information due to excessive weighting. This fusion mechanism ensures that the stable expression of the key posture node in each modality image can still be accurately extracted under interference conditions (such as strong light variation), and finally outputs a shooting posture fusion feature map with strong cross-modal consistency and clear semantics.

[0095] It should be noted that the scheme of the present application synchronously collects visible light images and infrared thermal images, realizes joint recognition of shooting posture nodes under multi-modal information, and uses multi-scale convolution to extract spatial distribution features, effectively capturing posture details at different scales. In the context of strong light interference, a fuzzy registration method driven by distribution difference loss is used to associate cross-modal node elements, further based on attribution confidence to construct pixel-level constraint conditions, enhance the accuracy of node matching between images, and finally introduce a self-attention mechanism and hierarchical constraint fusion strategy, while considering spatial consistency and semantic continuity, thereby significantly improving the robustness and precision of shooting posture recognition in complex outdoor environments.

[0096] On the other hand, in some embodiments, the present application provides an image recognition-based outdoor shooting assistance system, which comprises an image recognition unit, as described above with reference to Figure 4 The figure is a structural schematic diagram of an image recognition unit according to some embodiments of the present application, which comprises an acquisition module 401, a processing module 402 and an execution module 403, which are described as follows:

[0097] The acquisition module 401 is mainly used for synchronously collecting visible light images and infrared thermal images of a shooter's shooting posture in an outdoor shooting environment.

[0098] The processing module 402 is used for performing multi-scale convolution on the visible light images and the infrared thermal images, respectively, extracting spatial distribution features of shooting posture nodes in the visible light images and the infrared thermal images, and further determining the distribution difference loss of the same shooting posture node between different heterogeneous images.

[0099] The processing module 402 is also used for fuzzy registration of element labels of shooting posture nodes in different heterogeneous images based on all distribution difference losses when the images are interfered by strong light during image adoption, obtaining attribution confidence of shooting posture nodes in different heterogeneous images, and further determining pixel constraint conditions of shooting posture nodes in different heterogeneous images under interference suppression according to all attribution confidences.

[0100] The execution module 403 is mainly used for generating cross-modal fusion attention of shooting posture nodes between different heterogeneous images based on a self-attention mechanism, performing hierarchical constraint fusion of spatial distribution features of shooting posture nodes in different heterogeneous images by the cross-modal fusion attention and the pixel constraint conditions of shooting posture nodes, and obtaining fusion features of shooting postures.

[0101] In addition, the present application further provides a computer device, comprising a memory and a processor, the memory stores codes, and the processor is configured to acquire the codes and execute the image recognition method of the shooting posture.

[0102] In some embodiments, with reference to Figure 5 The figure is a structural schematic diagram of a computer device for implementing the image recognition method of the shooting posture according to some embodiments of the present application. The image recognition method of the shooting posture in the above embodiments can be implemented by the computer device shown in the figure, which comprises at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504. Figure 5 The processor 501 can be a general central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0103] The processor 501 can be a general central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0104] The communication bus 502 can be used to transmit information between the above components.

[0105] The memory 503 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 503 can exist independently and be connected to the processor 501 through the communication bus 502. The memory 503 can also be integrated with the processor 501.

[0106] The memory 503 is configured to store a program code for implementing the scheme of the present application, and the processor 501 is configured to execute the program code stored in the memory 503. The program code can include one or more software modules. The image recognition method of the shooting posture in the above-mentioned embodiments can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.

[0107] The communication interface 504 is configured to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc., using any transceiver-like device.

[0108] In a specific implementation, as an example, the computer device can include a plurality of processors, each of which can be a single-CPU processor or a multi-CPU processor. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0109] The computer device described above can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of the computer device.

[0110] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the image recognition method of the shooting posture.

[0111] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to cover all changes and modifications falling within the scope of the present application.

[0112] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. An image recognition method of a shooting posture, for an outdoor shooting auxiliary system to recognize a shooting posture of an outdoor shooter, characterized in that, The method comprises the following steps: Synchronously collecting a visible light image and an infrared thermal image of a shooting posture of a shooter in an outdoor shooting environment; Respectively performing multi-scale convolution on the visible light image and the infrared thermal image to extract spatial distribution features of shooting posture nodes in the visible light image and the infrared thermal image, and then determining distribution difference losses of a same shooting posture node between different heterogeneous images; When the image acquisition process is disturbed by strong light, fuzzy registration is performed on element labels of the shooting posture nodes in different heterogeneous images based on all the distribution difference losses to obtain attribution confidence of the shooting posture nodes in different heterogeneous images, and then pixel constraint conditions of the shooting posture nodes in different heterogeneous images in interference suppression are determined according to all the attribution confidence; Cross-modal fusion attention of the shooting posture nodes between different heterogeneous images is generated based on a self-attention mechanism, spatial distribution features of the shooting posture nodes in different heterogeneous images are subjected to hierarchical constraint fusion based on the cross-modal fusion attention and the pixel constraint conditions of the shooting posture nodes, and fusion features of the shooting posture are obtained; The fuzzy registration of the element labels of the shooting posture nodes in different heterogeneous images based on all the distribution difference losses to obtain the attribution confidence of the shooting posture nodes in different heterogeneous images specifically comprises: The fuzzy membership degrees of the shooting posture nodes between different heterogeneous images are determined from all the distribution difference losses; Label confidence between the shooting posture nodes is obtained by fuzzy registration reasoning of the element labels of the shooting posture nodes in different heterogeneous images according to all the fuzzy membership degrees, and the label confidence is an index for measuring the matching reliability degree between the labels of the shooting posture nodes in different modal images; The attribution confidence of the shooting posture nodes in different heterogeneous images is determined through all the label confidence.

2. The method of claim 1, wherein, The extraction of the spatial distribution features of the shooting posture nodes in the visible light image and the infrared thermal image, and then the determination of the distribution difference losses of a same shooting posture node between different heterogeneous images specifically comprises: A convolutional neural network module with multi-scale receptive field is constructed; The spatial distribution features of the shooting posture nodes in the visible light image and the infrared thermal image are extracted through the convolutional neural network module; A node label vector of the shooting posture nodes in the visible light image is determined according to the spatial distribution features corresponding to the visible light image; A node label vector of the shooting posture nodes in the infrared thermal image is determined according to the spatial distribution features corresponding to the infrared thermal image; The distribution difference losses of a same shooting posture node between different heterogeneous images are determined through all the node label vectors.

3. The method of claim 1, wherein, The determination of the pixel constraint conditions of the shooting posture nodes in different heterogeneous images in interference suppression according to all the attribution confidence specifically comprises: Multi-objective collaborative constraint modeling is performed on interference suppression parameters of pixels of the shooting posture nodes in different heterogeneous images according to all the attribution confidence to obtain a pixel constraint condition set; Optimization of the interference suppression parameters is performed on the pixel constraint condition set based on a multi-objective optimization algorithm to obtain optimal suppression parameters of the pixels of the shooting posture nodes in different heterogeneous images. The pixel constraints of the shooting posture node for different heterogeneous images during interference suppression are determined by using all the optimal pixel suppression parameters.

4. The method of claim 1, wherein, The cross-modal fusion attention mechanism for generating shooting pose nodes between different heterogeneous images based on self-attention specifically includes: Attention weight matrices for shooting posture nodes in different heterogeneous images are generated based on a self-attention mechanism; By fusing the feature attention of shooting posture nodes across different heterogeneous images using all attention weight matrices, cross-modal fused attention of shooting posture nodes across different heterogeneous images is obtained.

5. The method of claim 1, wherein, The visible light image is captured by a camera.

6. The method of claim 1, wherein, The infrared thermal image is acquired using an infrared thermal imager.

7. An image recognition-based outdoor shooting assistance system, comprising an image recognition unit that performs image recognition using the method according to any one of claims 1 to 6, characterized in that, The image recognition unit includes: The acquisition module is used to simultaneously acquire visible light and infrared thermal images of the shooter's shooting posture in an outdoor shooting environment; The processing module is used to perform multi-scale convolution on the visible light image and the infrared thermal image respectively, extract the spatial distribution features of the shooting posture node in the visible light image and the infrared thermal image, and then determine the distribution difference loss of the same shooting posture node in different heterogeneous images; The processing module is also used to perform fuzzy registration of the element labels of shooting posture nodes in different heterogeneous images based on all distribution difference losses when the image is subjected to strong light interference during image acquisition, to obtain the attribution confidence of shooting posture nodes in different heterogeneous images, and then determine the pixel constraint conditions of shooting posture nodes in different heterogeneous images when interference is suppressed based on all attribution confidence. The execution module is used to generate cross-modal fusion attention for shooting posture nodes in different heterogeneous images based on a self-attention mechanism. The spatial distribution features of shooting posture nodes in different heterogeneous images are hierarchically constrained and fused by the cross-modal fusion attention and the pixel constraints of the shooting posture nodes to obtain the fused features of the shooting posture.

8. A computer device comprising a memory and a processor, the memory storing code, wherein, The processor is configured to acquire the code and execute the image recognition method for shooting posture as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. When the computer program is executed by the processor, it implements the image recognition method for shooting posture as described in any one of claims 1 to 6.

Images