Intelligent aiming and shooting method and system
Through image acquisition equipment and deep learning models, combined with Kalman filtering, real-time recognition and precise strikes of targets are achieved, and the problem of difficulty in simulating dynamic scenes and precise recognition of traditional training systems is solved, improving the accuracy and efficiency of military training.
Patent Information
- Application Number
- CN202411140836.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-20
AI Technical Summary
Traditional military training systems are difficult to simulate dynamic changes and complex scenes in real battlefield environments, and cameras are difficult to achieve accurate identification and tracking of targets. The existing technology lacks real-time identification and intelligent targeting and strike solutions.
Image acquisition equipment is used to obtain shooting scene videos and images, use YOLO v4 model for object detection and classification, combine color features and texture features for target type recognition, use OpenPose to extract target key points, and use long and short-term memory network to perform action recognition, and use Kalman filter predicts the target position and motion trajectory, calculates shooting timing and ballistics, and dynamically adjusts shooting parameters to achieve accurate strikes.
Real-time identification and distinction of site targets is achieved, and prediction is dynamically adjusted to ensure continuous and accurate target tracking, ensuring that the launch object meets the target at the optimal time and path, improving combat accuracy and efficiency.
Smart Images

Figure CN118657933B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent aiming and shooting, in particular to an intelligent aiming and shooting method and system. Background Art
[0002] In modern military training and simulated confrontation systems, accurate target recognition and strike capabilities are key. Traditional training systems usually rely on static targets and preset action patterns, which makes it difficult to simulate dynamic changes and complex scenes in real battlefield environments. In the prior art, cameras are usually used to monitor the site in real time, but it is difficult to accurately identify and track the target by relying solely on the visual capture capability of the camera. The introduction of the pan / tilt allows the camera to monitor in all directions without blind spots, but to achieve accurate strikes on targets, it is also necessary to combine advanced algorithms and sophisticated control systems. Therefore, a solution that can collect, identify, and intelligently aim at targets in real time is of great significance. Summary of the invention
[0003] In order to solve the above technical problems in the prior art, the present invention proposes an intelligent aiming and shooting method and system to solve the above technical problems.
[0004] According to a first aspect of the present invention, an intelligent aiming and shooting method comprises:
[0005] S1: Obtain the video and image of the shooting scene through the image acquisition device, and use the target recognition algorithm to identify the target attributes and relative position information. The target attributes include target type recognition, posture recognition and action recognition. The relative position information includes the relative distance and direction from the shooting position, and the actual position coordinates are calculated. The target recognition algorithm specifically includes:
[0006] S11: Target detection and classification using YOLO v4 model;
[0007] S12: Target type recognition through color and texture features;
[0008] S13: Use OpenPose or a deep learning model for pose estimation to extract the target’s key points, including shoulders, elbows, and knees, and use the key points to infer the target’s pose;
[0009] S14: Based on the results of posture recognition, action recognition is performed through long short-term memory network to infer the target's action;
[0010] S2: Predict the next position and motion trajectory of the target through Kalman filtering, calculate the shooting timing and trajectory according to the real-time position of the target, shoot according to the calculated trajectory and timing through the laser transmitter, and provide feedback and correction for each shot to dynamically adjust the shooting parameters.
[0011] In some specific embodiments, the relative distance is determined by the angle of the image acquisition device and the height of the target in the image. ,in, Indicates the actual height of a known or estimated target. Indicates the pixel value height of the target in the image, Indicates focal length; determines the azimuth of the target relative to the image acquisition device based on the horizontal position of the target in the image and the field of view of the image acquisition device ,in, represents the horizontal position of the target in the image, Indicates the width of the image. Indicates the field of view of the image acquisition device; the actual position coordinates of the target are calculated through the actual position and relative position information of the image acquisition device , , , ,in, Indicates the actual position coordinates of the image acquisition device.
[0012] In some specific embodiments, the YOLO v4 model includes a backbone network, a feature fusion network and a detection head, the backbone network is a deep convolutional neural network for extracting image features, the feature fusion network includes a feature pyramid network and a path aggregation grid, the detection head is used to predict the bounding box, target category and confidence, and the loss function of the YOLO v4 model includes a bounding box regression loss , confidence loss and classification loss , the total loss function ,in, , , ,in, Indicates the side length of the grid; represents the number of bounding boxes predicted for each grid; Indicates when The grid When a bounding box contains an object, the value is 1, otherwise it is 0; and represents the center coordinates of the predicted bounding box; and Represents the true bounding box center coordinates; and represents the predicted bounding box width and height; and Represents the true bounding box width and height; Represents the confidence of the predicted bounding box; Represents the true bounding box confidence; Indicates when The grid When there is no object in the bounding box, the value is 1, otherwise it is 0; Represents the predicted class probability; Represents the true class probability.
[0013] In some specific embodiments, target type recognition specifically includes: using the YOLO v4 model to detect and classify the target, and outputting the bounding box and category label of each target; using the Mask R-CNN model to perform semantic segmentation to separate the target part in the image from the background, and the target part includes the epaulettes of the target; converting the RGB image into HSV space, and extracting the color features of the epaulettes by setting a threshold; and judging whether the target belongs to the red side or the blue side according to the color features and position information through attribute classification.
[0014] In some specific embodiments, the Mask R-CNN model includes a backbone network, a region proposal network, ROIAlign, a classification branch, and a mask branch. ROI Align crops the ROI from the feature map extracted by the backbone network according to the candidate bounding box, and aligns it to the feature map of a fixed size; the mask branch is applied to each ROI to generate a pixel-level binary mask to indicate the precise position of the target in the image.
[0015] In some specific embodiments, the long short-term memory network includes an input layer, a hidden layer and an output layer, the input layer receives the key sequence of the target, the hidden layer includes multiple LSTM units, and the state update of each LSTM unit is realized through a forget gate, an input gate, a cell state and an output gate, wherein the forget gate state update formula is: , represents the output of the forget gate, for Activation function, represents the weight matrix of the forget gate, represents the hidden state at the previous moment, Represents the input data at the current moment, Represents the bias of the forget gate; the input gate state update formula includes , , represents the output of the input gate, represents the weight matrix of the input gate, represents the bias of the input gate, represents the new candidate memory cell state, represents the weight matrix of candidate memory cells, represents the bias of the candidate memory cell; the cell state update formula is , Indicates the cell state at the current moment, Represents the cell state at the previous moment; the output gate state update formula includes , , represents the output of the output gate, represents the weight matrix of the output gate, represents the bias of the output gate, Represents the hidden state at the current moment; the output layer receives the output of the hidden layer to generate the final prediction result of the target action category. The calculation formula of the output layer is , represents the weight matrix of the output layer, represents the bias of the output layer.
[0016] In some specific embodiments, predicting the next position and motion trajectory of the target by Kalman filtering in S2 specifically includes: a multi-target tracker based on Kalman filtering tracks the position change of the target in consecutive video frames and predicts the motion trajectory, wherein the prediction step includes: , ; The update steps include: , , ,in, represents the predicted state estimate, represents the predicted state covariance, represents the state transfer matrix, represents the control input matrix, represents the control input, represents the process noise covariance, represents the Kalman gain, represents the measurement matrix, represents the measurement noise covariance, represents the updated state estimate, represents the updated state covariance.
[0017] In some specific embodiments, the calculation of the shooting timing and trajectory according to the real-time position of the target specifically includes: predicting the position of the target at a certain point in the future according to the current position and speed of the target; determining the speed of the projectile and its trajectory after launch, and the shooting timing satisfies that the projectile can intersect with the target at the same position at the time point. The specific formula is as follows: The motion model of the target is expressed as: the target at time The position is represented by , ,in, Indicates that the target is at the initial time The horizontal position coordinates at time represents the velocity component of the target in the horizontal direction, Indicates that the target is at the initial time The vertical position coordinate at time, represents the velocity component of the target in the vertical direction, The target's motion direction angle is the angle between the target's motion direction and the horizontal axis. The motion model of the projectile is expressed as follows: the projectile moves from position Launch, launch at time The position is represented by , ,in, Indicates that the projectile is at the initial moment The horizontal position coordinates at time Indicates that the projectile is at the initial moment The vertical position coordinate at time, represents the velocity component of the projectile in the horizontal direction, represents the velocity component of the projectile in the vertical direction, Indicates the launch angle of the missile, which is the angle between the missile's direction of motion and the horizontal axis. and Determine the launch time and launch angle To satisfy both and .
[0018] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.
[0019] According to a third aspect of the present invention, an intelligent aiming and shooting system is proposed, comprising:
[0020] Target recognition unit: configured to acquire video and images of shooting scenes through image acquisition equipment, and identify target attributes using target recognition algorithms, which include target type recognition, posture recognition, and action recognition; the target recognition algorithm specifically includes: using the YOLO v4 model for target detection and classification; identifying target types through color features and texture features; extracting key points of the target using OpenPose or a deep learning model for posture estimation, which include shoulders, elbows, and knees, and inferring the posture of the target using the key points; based on the results of posture recognition, performing action recognition through a long short-term memory network to infer the target's actions;
[0021] Target prediction and shooting unit: It is configured to predict the next position and motion trajectory of the target through Kalman filtering, calculate the shooting timing and trajectory according to the real-time position of the target, shoot according to the calculated trajectory and timing through the laser transmitter, and provide feedback and correction for each shot to dynamically adjust the shooting parameters.
[0022] The present invention proposes an intelligent aiming and shooting method and system, which has the following technical effects:
[0023] The present invention can instantly identify targets on the field, including distinguishing between enemies and friends, posture analysis (such as standing or lying), action prediction (such as preparing to shoot or moving), etc., and classify based on the color of the epaulettes or other features; it can predict the future position and movement trajectory of the target, adjust the prediction in real time, and ensure continuous and accurate target tracking; by dynamically adjusting the launch parameters, it can ensure that the projectile meets the target at the optimal time and path, thereby achieving a precise strike effect. The angle of the laser launcher can also be controlled by the pan-tilt system to achieve precise strikes on the target. The projectile (such as a digital simulation bullet) accurately hits the target according to the predicted target position and movement trajectory, improving the accuracy and efficiency of combat. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and together with the description are used to explain the principles of the present invention. Other embodiments and many expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:
[0025] Figure 1 is a flow chart of an intelligent aiming and shooting method according to an embodiment of the present application;
[0026] Figure 2 is a flowchart of target identification of a specific embodiment of the present application;
[0027] Figure 3 is a framework diagram of an intelligent aiming and shooting method system according to an embodiment of the present application;
[0028] Figure 4 It is a framework diagram of an intelligent aiming and shooting method system of a specific embodiment of the present application;
[0029] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION
[0030] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0032] Figure 1 FIG. 2 shows a flow chart of an intelligent aiming and shooting method according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0033] S1: Obtaining the video and image of the shooting scene through the image acquisition device, and using the target recognition algorithm to identify the target attributes and relative position information, the target attributes include target type recognition, posture recognition and action recognition, the relative position information includes the relative distance and direction from the shooting position, and the actual position coordinates are calculated;
[0034] In a specific embodiment, Figure 2 A flowchart of a target recognition algorithm of a specific embodiment of the present application is shown as follows: Figure 2 As shown in the figure, the basic information configuration specifically includes:
[0035] S11: Use the YOLO v4 model for object detection and classification. The YOLO v4 model includes a backbone network, a feature fusion network, and a detection head. The backbone network is a deep convolutional neural network used to extract image features. The feature fusion network includes a feature pyramid network and a path aggregation grid. The detection head is used to predict bounding boxes, object categories, and confidence levels.
[0036] In a specific embodiment, the loss function of the YOLO v4 model includes the bounding box regression loss , confidence loss and classification loss , the total loss function ,in, , , ,in, Indicates the side length of the grid; represents the number of bounding boxes predicted for each grid; Indicates when The grid When a bounding box contains an object, the value is 1, otherwise it is 0; and represents the center coordinates of the predicted bounding box; and Represents the true bounding box center coordinates; and represents the predicted bounding box width and height; and Represents the true bounding box width and height; Represents the confidence of the predicted bounding box; Represents the true bounding box confidence; Indicates when The grid When there is no object in the bounding box, the value is 1, otherwise it is 0; Represents the predicted class probability; Represents the true class probability.
[0037] S12: Identify the target type through color features and texture features. Specifically, the color histogram and HOG features (Histogram of Oriented Gradients) can be used for classification.
[0038] In a specific embodiment, target type recognition specifically includes: using the YOLO v4 model to detect and classify the target, and outputting the bounding box and category label of each target; using the Mask R-CNN model to perform semantic segmentation to separate the target part in the image from the background, and the target part includes the epaulettes of the target; converting the RGB image into HSV space, and extracting the color features of the epaulettes by setting a threshold; and judging whether the target belongs to the red side or the blue side according to the color features and position information through attribute classification.
[0039] In a specific embodiment, the Mask R-CNN model includes a backbone network (using ResNet or a higher-level backbone network, such as ResNet-101, for extracting features from an image), a region proposal network (generating candidate target bounding boxes), ROI Align (region of interest alignment), a classification branch (for predicting the category label of the target) and a mask branch (based on the bounding box, generating an accurate mask for each target for pixel-level semantic segmentation). ROI Align crops the ROI from the feature map extracted by the backbone network according to the candidate bounding box, and aligns it to a feature map of a fixed size; the mask branch is applied to each ROI to generate a pixel-level binary mask to indicate the precise position of the target in the image.
[0040] S13: Use OpenPose or a deep learning model for pose estimation to extract the target’s key points, including shoulders, elbows, and knees, and use the key points to infer the target’s pose.
[0041] S14: Based on the results of posture recognition, action recognition is performed through the long short-term memory network to infer the target's action.
[0042] In a specific embodiment, the long short-term memory network includes an input layer, a hidden layer and an output layer. The input layer receives the key sequence of the target, and the hidden layer includes multiple LSTM units. The state update of each LSTM unit is realized through a forget gate, an input gate, a cell state and an output gate. The specific implementation method is as follows: The forget gate state update formula is , represents the output of the forget gate, for Activation function, represents the weight matrix of the forget gate, represents the hidden state at the previous moment, Represents the input data at the current moment, Represents the bias of the forget gate.
[0043] The input gate state update formula includes , , represents the output of the input gate, represents the weight matrix of the input gate, represents the bias of the input gate, represents the new candidate memory cell state, represents the weight matrix of candidate memory cells, represents the bias of candidate memory cells.
[0044] The cell state update formula is: , Indicates the cell state at the current moment, Indicates the cell state at the previous moment.
[0045] The output gate state update formula includes , , represents the output of the output gate, represents the weight matrix of the output gate, represents the bias of the output gate, Indicates the hidden state at the current moment.
[0046] The output layer receives the output of the hidden layer to generate the final prediction result of the target action category. The calculation formula of the output layer is , represents the weight matrix of the output layer, Represents the bias of the candidate output layer. The action category output by the LSTM network can identify the target's action (such as preparing to shoot, moving position, etc.).
[0047] In a specific embodiment, the relative distance is determined by the angle of the image acquisition device and the height of the target in the image. ,in, Indicates the actual height of a known or estimated target. Indicates the pixel value height of the target in the image, Indicates focal length; determines the azimuth of the target relative to the image acquisition device based on the horizontal position of the target in the image and the field of view of the image acquisition device ,in, represents the horizontal position of the target in the image, Indicates the width of the image. Indicates the field of view of the image acquisition device; the actual position coordinates of the target are calculated through the actual position and relative position information of the image acquisition device , , , ,in, Indicates the actual position coordinates of the image acquisition device. Through the above steps and algorithms, the system can achieve real-time recognition, position calculation and precise strike of the target.
[0048] S2: Predict the next position and motion trajectory of the target through Kalman filtering, calculate the shooting timing and trajectory according to the real-time position of the target, shoot according to the calculated trajectory and timing through the laser transmitter, and provide feedback and correction for each shot to dynamically adjust the shooting parameters.
[0049] In a specific embodiment, a multi-target tracker based on Kalman filtering tracks the position change of the target in consecutive video frames and predicts the motion trajectory, wherein the prediction step includes: , ; The update steps include: , , ,in, represents the predicted state estimate, Represents the time step The posterior state estimate (i.e., the state estimate after combining the observed data) is represents the predicted state covariance, Represents the time step The posterior covariance estimate of (representing the uncertainty of the state after combining the observed data), represents the state transfer matrix, represents the control input matrix, represents the control input, represents the process noise covariance, represents the Kalman gain, represents the measurement matrix, represents the measurement noise covariance, represents the updated state estimate, represents the updated state covariance.
[0050] In a specific embodiment, the position of the target at a certain point in the future is predicted based on the current position and speed of the target; the speed of the projectile and its trajectory after launch are determined, and the shooting timing satisfies that the projectile can intersect with the target at the same position at the time point. The specific formula is as follows: The motion model of the target is expressed as: the target at time The position is represented by , ,in, Indicates that the target is at the initial time The horizontal position coordinates at time represents the velocity component of the target in the horizontal direction, Indicates that the target is at the initial time The vertical position coordinate at time, represents the velocity component of the target in the vertical direction, The target's motion direction angle is the angle between the target's motion direction and the horizontal axis. The motion model of the projectile is expressed as follows: the projectile moves from position Launch, launch at time The position is represented by , ,in, Indicates that the projectile is at the initial moment The horizontal position coordinates at time Indicates that the projectile is at the initial moment The vertical position coordinate at time, represents the velocity component of the projectile in the horizontal direction, represents the velocity component of the projectile in the vertical direction, Indicates the launch angle of the missile, which is the angle between the missile's direction of motion and the horizontal axis. and Determine the launch time and launch angle To satisfy both and .
[0051] Figure 3 FIG. 1 shows a system architecture diagram of an intelligent aiming and shooting method according to an embodiment of the present application. Figure 3As shown, the system includes a target recognition unit 301 and a target prediction and shooting unit 302, wherein the target recognition unit 301 is configured to obtain videos and images of shooting scenes through an image acquisition device, and to identify target attributes using a target recognition algorithm, wherein the target attributes include target type recognition, posture recognition, and action recognition; the target recognition algorithm specifically includes: using a YOLO v4 model to detect and classify targets; identifying target types through color features and texture features; extracting key points of the target using an OpenPose or a deep learning model for posture estimation, wherein the key points include shoulders, elbows, and knees, and using the key points to infer the posture of the target; based on the result of posture recognition, performing action recognition through a long short-term memory network to infer the action of the target; the target prediction and decision unit 302 is configured to predict the next position and motion trajectory of the target through a Kalman filter, calculate the shooting timing and trajectory according to the real-time position of the target, shoot according to the calculated trajectory and timing through a laser transmitter, and provide feedback and correction for each shooting to dynamically adjust the shooting parameters.
[0052] Figure 4 The framework diagram of the intelligent aiming and shooting method system of a specific embodiment of the present application is shown, and the system includes a target 401, a motion platform 402, a server 403, a camera 404, a laser transmitter 405 and a target drone 406. Among them, the target 401 is the object that the system needs to identify, track and strike; the camera 404 and the laser transmitter 405 are both installed on the motion platform 402, and the movement of the motion platform 402 realizes all-round monitoring and precise strikes on the site; the server 403 is the core processing unit, which is responsible for receiving the video data transmitted by the camera, running the aforementioned target recognition and prediction algorithm, and controlling the movement of the motion platform and the laser transmitter; the camera 404 is used to collect the scene of the site in real time, and transmit the video data to the server for processing; the laser transmitter 405 launches a laser simulation bullet to strike the target according to the instructions of the server; the target drone 406: as a target in the simulated confrontation, it can receive the attack signal and provide feedback.
[0053] The system workflow is as follows: the camera 404 collects the scene in real time and transmits the video data to the server 403. The server 403 runs the target recognition and prediction algorithm to identify the attributes, posture, action and real-time position of the target 401, and predict the next trajectory and intention of the target. According to the prediction results, the server 403 controls the motion pan / tilt 402 to adjust the angle of the camera 404 and the laser transmitter 405 to achieve real-time tracking and precise strikes on the target 401. The target drone 406 can also participate in the confrontation simulation to provide target response and feedback.
[0054] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0055] like Figure 5 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for system operation are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0056] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.
[0057] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.
[0058] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0059] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0060] The modules involved in the embodiments of the present application may be implemented by software or hardware.
[0061] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist alone and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: obtains the video and image of the shooting scene through the image acquisition device, and uses the target recognition algorithm to identify the target attributes, and the target attributes include target type recognition, posture recognition and action recognition; the target recognition algorithm specifically includes: using the YOLO v4 model for target detection and classification; using color features and texture features to identify the target type; using OpenPose or a deep learning model for posture estimation to extract the key points of the target, the key points include shoulders, elbows and knees, and using the key points to infer the posture of the target; based on the results of posture recognition, the action recognition is performed through the long short-term memory network to infer the action of the target; the next position and motion trajectory of the target are predicted through the Kalman filter, the shooting timing and trajectory are calculated according to the real-time position of the target, and the laser transmitter is used to shoot according to the calculated trajectory and timing, and each shot is fed back and corrected, and the shooting parameters are dynamically adjusted.
[0062] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. An intelligent aiming and shooting method, It is characterized in that include: S1: Obtain the video and image of the shooting scene through the image acquisition device, and use the target recognition algorithm to identify the target attributes and relative position information, the target attributes include target type recognition, posture recognition and action recognition, the relative position information includes the relative distance and direction from the shooting position, and calculate the actual position coordinates; the target recognition algorithm specifically includes: S11: Target detection and classification using YOLO v4 model; S12: Target type recognition through color and texture features; S13: extracting key points of the target using OpenPose or a deep learning model for posture estimation, where the key points include shoulders, elbows, and knees, and inferring the posture of the target using the key points; S14: Based on the result of the posture recognition, perform action recognition through a long short-term memory network to infer the action of the target; S2: predict the next position and motion trajectory of the target through Kalman filtering, calculate the shooting timing and trajectory according to the real-time position of the target, shoot according to the calculated trajectory and timing through the laser transmitter, and provide feedback and correction for each shooting to dynamically adjust the shooting parameters; The YOLO v4 model includes a backbone network, a feature fusion network and a detection head. The backbone network is a deep convolutional neural network for extracting image features. The feature fusion network includes a feature pyramid network and a path aggregation grid. The detection head is used to predict bounding boxes, target categories and confidence levels. The loss function of the YOLO v4 model includes a bounding box regression loss Loss bbox , confidence loss Loss conf And classification loss Loss cls , the total loss function Loss = Loss bbox +Loss conf +Loss cls ,in, Where s represents the side length of the grid; B represents the number of bounding boxes predicted for each grid; Indicates that when the jth bounding box of the i-th grid contains an object, the value is 1, otherwise it is 0; x i and i Represents the predicted bounding box center coordinates; and represents the actual bounding box center coordinates; w i and h i represents the predicted bounding box width and height; and represents the actual bounding box width and height; C i Represents the confidence of the predicted bounding box; Represents the true bounding box confidence; Indicates that when there is no target in the jth bounding box of the i-th grid, the value is 1, otherwise it is 0; p i (c) represents the predicted category probability; Represents the true category probability; The relative distance is determined by the angle of the image acquisition device and the height of the target in the image. Among them, H real Indicates the actual height of the known or estimated target, H image represents the pixel value height of the target in the image, and f represents the focal length; according to the horizontal position of the target in the image and the field of view of the image acquisition device, the azimuth angle of the target relative to the image acquisition device is determined Among them, x image represents the horizontal position of the target in the image, W image represents the width of the image, α represents the field of view of the image acquisition device; the actual position coordinates (X target , Y target , Z target ), X target =X cam +d cosθ,Y target =Y cam +d sinθ,Z target =Z cam , where (X cam , Y cam , Z cam ) represents the actual position coordinates of the image acquisition device.
2. The intelligent aiming and shooting method according to claim 1, It is characterized in that The target type recognition specifically includes: using the YOLO v4 model to perform target detection and classification, and outputting the bounding box and category label of each target; using the Mask R-CNN model to perform semantic segmentation, and separating the target part in the image from the background, and the target part includes the epaulettes of the target; converting the RGB image into HSV space, and extracting the color features of the epaulettes by setting a threshold; and judging whether the target belongs to the red side or the blue side according to the color features and position information through attribute classification.
3. The intelligent aiming and shooting method according to claim 2, It is characterized in that The Mask R-CNN model includes a backbone network, a region proposal network, ROIAlign, a classification branch, and a mask branch. The ROIAlign cuts out the ROI from the feature map extracted by the backbone network according to the candidate bounding box and aligns it to a feature map of a fixed size. The mask branch is applied to each of the ROIs to generate a pixel-level binary mask representing the precise location of the object in the image.
4. The intelligent aiming and shooting method according to claim 1, It is characterized in that The long short-term memory network includes an input layer, a hidden layer and an output layer, wherein the input layer receives the key sequence of the target, the hidden layer includes a plurality of LSTM units, and the state update of each LSTM unit is realized through a forget gate, an input gate, a cell state and an output gate, wherein the forget gate state update formula is f t =σ(W f ·[h t-1 , x t ]+b f ), f t represents the output of the forget gate, σ is the Sigmoid activation function, W f represents the weight matrix of the forget gate, h t-1 represents the hidden state at the previous moment, x t represents the input data at the current moment, b f Represents the bias of the forget gate; the input gate state update formula includes i t =σ(W i ·[h t-1 , x t ]+b i ), i t represents the output of the input gate, W i represents the weight matrix of the input gate, b i represents the bias of the input gate, represents the new candidate memory cell state, W C represents the weight matrix of candidate memory cells, b c represents the bias of the candidate memory cell; the cell state update formula is C t Indicates the cell state at the current moment, C t-1 Represents the cell state at the previous moment; the output gate state update formula includes o t =σ(W o ·[h t-1 , x t ]+b o ), h t =o t *tanh(C t ), o t represents the output of the output gate, W o represents the weight matrix of the output gate, b o represents the bias of the output gate, h t Represents the hidden state at the current moment; the output layer receives the output of the hidden layer to generate the final prediction result of the target action category, and the calculation formula of the output layer y t =softmax(W y ·h t +b y ), W y represents the weight matrix of the output layer, b y Represents the bias of the output layer.
5. The intelligent aiming and shooting method according to claim 1, It is characterized in that Predicting the next position and motion trajectory of the target by Kalman filtering in S2 specifically includes: a multi-target tracker based on Kalman filtering tracks the position change of the target in continuous video frames and predicts the motion trajectory, wherein the prediction step includes: P k|k-1 =FP k-1|k-1 F T +Q; the update steps include: K k =P k|k-1 H T (HP k|k-1 H T +R) -1 , P k|k =(IK k H)P k|k-1 ,in, represents the predicted state estimate, P k|k-1 represents the predicted state covariance, F represents the state transfer matrix, B represents the control input matrix, u k represents the control input, Q represents the process noise covariance, K k represents the Kalman gain, H represents the measurement matrix, R represents the measurement noise covariance, represents the updated state estimate, P k|k represents the updated state covariance.
6. The intelligent aiming and shooting method according to claim 1, It is characterized in that The calculation of the shooting timing and trajectory according to the real-time position of the target specifically includes: predicting the position of the target at a certain time point in the future according to the current position and speed of the target; determining the speed of the projectile and its trajectory after launch, and the shooting timing satisfies that the projectile can intersect with the target at the same position at the time point. The specific formula is as follows: The motion model of the target is expressed as: The position of the target at time t is expressed as x T (t) = x T (0)+v T cos(θ T )t,y T (t) = y T (0)+v T sin(θ T )t, where x T (0) represents the horizontal position coordinate of the target at the initial time t = 0, v T cos(θ T ) represents the velocity component of the target in the horizontal direction, y T (0) represents the vertical position coordinate of the target at the initial time t = 0, v T sin(θ T ) represents the velocity component of the target in the vertical direction, θ T The target's moving direction angle is the angle between the target's moving direction and the horizontal axis. The motion model of the projectile is expressed as follows: the projectile moves from position (x F (0), y F (0)) is emitted, and its position at time t after emission is represented by x F (t) = x F (0)+v F cos(θ F )t,y F (t) = y F (0)+v F sin(θ F )t, where x F (0) represents the horizontal position coordinate of the launch object at the initial time t = 0, y F (0) represents the vertical position coordinate of the projectile at the initial time t = 0, v F cos(θ F ) represents the velocity component of the projectile in the horizontal direction, v F sin(θ F ) represents the velocity component of the projectile in the vertical direction, θ F Indicates the launch angle of the missile, which is the angle between the missile's direction of motion and the horizontal axis. T (t) = x F (t) and y T (t) = y F (t), determine the launch timing t and launch angle θ F To satisfy both and 7. A computer-readable storage medium having one or more computer programs stored thereon, It is characterized in that When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 6 is implemented.
8. An intelligent aiming and shooting system, It is characterized in that include: Target recognition unit: configured to acquire video and images of shooting scenes through image acquisition equipment, and use target recognition algorithm to identify target attributes and relative position information, wherein the target attributes include target type recognition, posture recognition and action recognition, and the relative position information includes relative distance and orientation from the shooting position, and the actual position coordinates are calculated; the target recognition algorithm specifically includes: using YOLO v4 model to detect and classify targets; using color features and texture features to identify target types; using OpenPose or a deep learning model for posture estimation to extract key points of the target, wherein the key points include shoulders, elbows and knees, and the posture of the target is inferred using the key points; based on the result of the posture recognition, action recognition is performed through a long short-term memory network to infer the action of the target, and the YOLO v4 model includes a backbone network, a feature fusion network and a detection head, wherein the backbone network is a deep convolutional neural network for extracting image features, the feature fusion network includes a feature pyramid network and a path aggregation grid, and the detection head is used to predict a bounding box, a target category and a confidence level; the loss function of the YOLO v4 model includes a bounding box regression loss Loss bbox , confidence loss Loss conf And classification loss Loss cls , the total loss function Loss = Loss bbox +Loss conf +Loss cls ,in, Where s represents the side length of the grid; B represents the number of bounding boxes predicted for each grid; Indicates that when the jth bounding box of the i-th grid contains an object, the value is 1, otherwise it is 0; x i and i Represents the predicted bounding box center coordinates; and represents the actual bounding box center coordinates; w i and h i represents the predicted bounding box width and height; and represents the actual bounding box width and height; C i Represents the confidence of the predicted bounding box; Represents the true bounding box confidence; Indicates that when there is no target in the jth bounding box of the i-th grid, the value is 1, otherwise it is 0; p i (c) represents the predicted category probability; Represents the true category probability; the relative distance is determined by the angle of the image acquisition device and the height of the target in the image, and the relative distance Among them, H real Indicates the actual height of the known or estimated target, H image represents the pixel value height of the target in the image, and f represents the focal length; according to the horizontal position of the target in the image and the field of view of the image acquisition device, the azimuth angle of the target relative to the image acquisition device is determined Among them, x image represents the horizontal position of the target in the image, W image represents the width of the image, α represents the field of view of the image acquisition device; the actual position coordinates (X target , Y target , Z target ), X target =X cam +d cosθ,Y target =Y cam +d sinθ,Z target =Z cam , where (X cam , Y cam , Z cam ) represents the actual position coordinates of the image acquisition device; Target prediction and shooting unit: configured to predict the next position and motion trajectory of the target through Kalman filtering, calculate the shooting timing and trajectory according to the real-time position of the target, shoot according to the calculated trajectory and timing through the laser transmitter, and provide feedback and correction for each shot to dynamically adjust the shooting parameters.
