A visual targeting localization method, apparatus, device, and medium

Abstract

The application provides a visual targeting positioning method, device, equipment and medium, the method comprises the following steps: collecting video stream data in the process of targeting shooting training and storing; pushing the collected video stream data to a streaming media server directly, and performing frame extraction processing on the video stream data by using the streaming media server; performing target object identification on the extracted image by using a trained neural network model, and generating a targeting shooting result according to the position of the identified target object; and displaying the targeting shooting result in a visual manner in real time. The application has the following advantages: compared with the traditional manual target scoring method, the subsequent identification process does not require manual intervention, so that the workload can be greatly reduced, the target scoring efficiency can be improved, and the safety can be ensured; compared with the existing method which needs to perform grayscale pretreatment operation on the video stream data, the delay can be effectively reduced, and the real-time performance of data display can be ensured.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a visual target positioning method, apparatus, device, and medium. Background Technology

[0002] Traditional target shooting training relies on manual target reporting. Specifically, after multiple shots are fired in succession, the shooting stops, and the judges come out from the cover near the target to observe the target surface at close range, or observe it from a distance using binoculars. The final target result is determined by the number and location of bullet holes on the target surface. However, the traditional manual target reporting method has problems such as large workload, low efficiency, and poor safety.

[0003] With the rapid development of modern technology, the military has increasingly higher requirements for the informatization of its bases, and at the same time, it has also placed higher demands on training services and management levels; using modern technology to achieve automatic target reporting in target practice has become a trend.

[0004] Existing technologies already contain solutions for automatic target reporting. For example, Chinese Patent Application No. CN201910599447.8, filed on July 4, 2019, discloses an automatic target reporting system, method, device, and medium. Specifically, the system includes: an image acquisition module for acquiring a first target image; an image recognition module for determining a first target area image based on the first target image, and determining a second target area image based on a second target image acquired before the first target image and adjacent to it; a bullet hole processing module for performing a subtraction operation between the gradient intensity image of the first target area image and the gradient intensity image of the second target area image to obtain a subtracted image, determining the bullet hole image corresponding to the first target area image based on the subtracted image, and determining the position coordinates of the bullet holes in the bullet hole image; and a broadcasting module for broadcasting the position coordinates of the bullet holes. However, existing technologies require preprocessing of video data, such as grayscale, resulting in significant delays in the entire data processing process and poor real-time performance. In view of the aforementioned problems, the inventors of this case conducted in-depth research on these problems, which led to the creation of this case. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a visual target positioning method, device, equipment and medium, which solves the problem that the existing automatic target reporting system requires preprocessing of video data, and the entire data processing process has a large delay, resulting in poor real-time performance.

[0006] In a first aspect, the present invention provides a visual target positioning method, the target positioning method comprising:

[0007] Collect and store video stream data during target practice training;

[0008] The acquired video stream data is directly pushed to the streaming media server, and the streaming media server is used to perform frame extraction processing on the video stream data.

[0009] The trained neural network model is used to identify target objects in the extracted images, and the target shooting results are generated based on the position of the identified target objects.

[0010] The results of target practice are displayed in real time in a visual manner.

[0011] Furthermore, the acquisition and storage of video stream data during the target practice training process specifically includes:

[0012] High-definition cameras are used to collect video stream data during target practice training.

[0013] A custom file system stores video stream data in a custom file system.

[0014] Stream encryption technology is used to encrypt the stored video stream data.

[0015] Furthermore, the custom file system, which stores video stream data in a custom file system, specifically includes:

[0016] Each disk on the file storage server is assigned an independent and unique disk index file, which contains the disk's unique serial number and slot information.

[0017] Configure the disk to store and retrieve files in fixed-size blocks;

[0018] After the video stream data is obtained from the disk in pass-through mode, the video stream data is divided into storage blocks of fixed size to obtain several small video stream files. At the same time, the offset and size of each small video stream file are calculated based on its position from the file header, and each small video stream file generates a unique offset index file based on its own offset. When accessing the video stream data, sequential reading and writing are performed based on the offset and size of each small video stream file.

[0019] Furthermore, the step of directly pushing the collected video stream data to the streaming media server specifically means: directly pushing the collected video stream data to the streaming media server via the RTMP protocol.

[0020] Furthermore, the step of using a streaming media server to perform frame extraction processing on the video stream data specifically includes:

[0021] During the training phase of the neural network model, when a target object is detected in the video stream data, the streaming media server obtains images of all frames in the video stream data according to the frame rate, filters out images that do not contain the target object, and uses the remaining images that contain the target object as the basic training material.

[0022] In the recognition stage after the neural network model is finalized, when a target object is detected in the video stream data, frames of the video stream data are extracted at preset time intervals.

[0023] Furthermore, the specific steps of identifying target objects in the extracted frames using a trained neural network model include:

[0024] Create neural network models using neural network algorithms;

[0025] A large number of images with target objects are obtained from the training materials as a dataset. The target objects in each image in the dataset are manually labeled, and the created neural network model is trained using the manually labeled images to obtain a trained neural network model.

[0026] During the recognition phase, the extracted frames of the image are input into a trained neural network model to automatically identify the target object.

[0027] Furthermore, the real-time visualization of the target shooting results specifically includes displaying the target shooting results through scrolling subtitles and voice broadcasting.

[0028] Secondly, the present invention provides a visual target positioning device, which includes a data acquisition and storage module, a streaming frame extraction module, an object recognition module, and a visualization display module.

[0029] The acquisition and storage module is used to acquire and store video stream data during target shooting training.

[0030] The push-stream frame extraction module is used to directly push the acquired video stream data to the streaming media server, and use the streaming media server to perform frame extraction processing on the video stream data.

[0031] The object recognition module is used to identify target objects in the frame-by-frame image using a trained neural network model, and generate target shooting results based on the position of the identified target objects.

[0032] The visualization module is used to display the target shooting results in a visual manner in real time.

[0033] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0034] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0035] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0036] 1. After acquiring video stream data during target practice, the system directly pushes the video stream data to a streaming media server. The server then performs frame extraction on the video stream data and uses a trained neural network model to automatically identify the extracted images without requiring additional image processing such as grayscale. The entire data processing latency is less than 1 second, resulting in minimal delay. Compared to traditional manual target reporting, aside from the initial manual annotation during training, the entire recognition process requires no manual intervention, significantly reducing workload, improving reporting efficiency, and ensuring safety. Compared to existing methods that require preprocessing of video stream data such as grayscale, this system effectively reduces latency and ensures real-time data display.

[0037] 2. By using a custom file system to store video stream data instead of a general file system, only personnel familiar with system programs and commands can obtain the required content from the custom file system. Other personnel cannot see file attributes and other related information under the Linux file system, thus providing excellent confidentiality.

[0038] 3. By setting up an independent and unique disk index file for each disk and generating a unique offset index file for each small video stream file based on its own offset, the specific storage location and disk information of each small video stream file can be accurately and quickly located through the above two layers of index files during implementation. This not only achieves the purpose of security, but also greatly improves the efficiency of read and write access.

[0039] 4. Stream encryption technology is used to directly encrypt the video stream data stored in each storage block on the disk. At the same time, the base64 encoding algorithm is used to encode the video stream data, which can ensure the security of data transmission.

[0040] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] Figure 1 This is a schematic diagram of the system framework corresponding to the present invention;

[0043] Figure 2 This is a flowchart illustrating the execution process of the target positioning method in Embodiment 1 of the present invention.

[0044] Figure 3 This is a schematic diagram of the target positioning device in Embodiment 2 of the present invention;

[0045] Figure 4 This is a schematic diagram of the electronic device in Embodiment 3 of the present invention;

[0046] Figure 5 This is a schematic diagram of the structure of the medium in Embodiment 4 of the present invention. Detailed Implementation

[0047] Before introducing specific embodiments, the system framework corresponding to the method in the embodiments of this application will be introduced first, such as... Figure 1 As shown, the system comprises approximately seven parts: a core switch, access switches, cameras, a file server, PCs and mobile devices, a streaming media server and an AI training server, a video decoder, and a video wall. Cameras are connected to the core switch via the access switch. The file server, PCs and mobile devices, the streaming media server and AI training server, and the video decoder are all connected to the core switch. The video wall is connected to the video decoder. The deployment of these parts forms a local area network (LAN), ensuring security. Specifically, the cameras are used to collect video stream data during target practice training; the core switch and access switches enable interconnection between devices; the file server stores the video stream data; the PCs and mobile devices can retrieve and play the stored video stream data from the file server when needed; the streaming media server performs frame extraction and other processing on the video stream data; the AI ​​training server trains the neural network model and uses the trained model for image recognition; the video decoder decodes the video stream data; and the video wall displays the target practice results in real time.

[0048] Example 1

[0049] This embodiment provides a visual target positioning method, such as Figure 2 As shown, the target positioning method includes:

[0050] Step S1: Collect and store video stream data during the target shooting training process;

[0051] Step S2: Push the acquired video stream data directly to the streaming media server, and use the streaming media server to perform frame extraction processing on the video stream data;

[0052] Step S3: The trained neural network model is used to identify target objects in the extracted images, and the target shooting results are generated based on the position of the identified target objects. The target shooting results include information such as whether the target was hit and the position of the hit location from the center of the bullseye.

[0053] Step S4: Display the target shooting results in a real-time, visual manner.

[0054] This invention employs a method where, after acquiring video stream data during target practice, the video stream data is directly pushed to a streaming media server. The streaming media server then performs frame extraction on the video stream data, and a trained neural network model automatically identifies the extracted images without requiring additional image processing such as grayscale. The entire data processing latency is less than 1 second, resulting in minimal delay. Compared to traditional manual target reporting methods, aside from the initial manual annotation required during training, the entire subsequent recognition process requires no manual intervention, thus significantly reducing workload, improving reporting efficiency, and ensuring safety. Compared to existing methods that require preprocessing of video stream data such as grayscale, this invention effectively reduces latency and ensures real-time data display.

[0055] In Embodiment 1 of the present invention, step S1, specifically including the acquisition and storage of video stream data during target practice and shooting training, includes:

[0056] High-definition cameras are used to collect video stream data during target practice training. The high-definition cameras are H265 / H264 high-definition cameras, and the image resolution needs to be high enough to clearly see the position of bullets or the base of smoke within a range of 300m. This can ensure the quality of the collected images and thus improve the accuracy of image recognition. In specific implementations, for example, the high-definition camera can be a high-speed infrared PTZ camera. This high-speed infrared PTZ camera uses a CMOS sensor of 1 / 1.8 inches or larger, with a pixel count of no less than 8 million (2688*1520), a zoom of no less than 22x, a focal length range of no less than 6.5-140mm, a frame rate of no less than 60 frames / s, and supports motion detection in at least 18*22 areas simultaneously. It can identify targets at least 500m away from the camera through infrared illumination. Alternatively, the high-definition camera can be an intelligent surveillance PTZ camera with a pixel count of 4 million or larger, an optical zoom of no less than 33x, a focal length range of no less than 6-198mm, a power zoom, and a TVL of no less than 1500 (resolution set to 2688×1520, frame rate set to 30fps, bit rate set to 2Mbps, RJ45 output).

[0057] This invention uses a custom file system to store video stream data instead of a general file system. Only personnel familiar with system programs and commands can access the required content from the custom file system. Others cannot see file attributes and other related information under the Linux file system, thus providing excellent confidentiality.

[0058] The stored video stream data is encrypted using streaming encryption technology (such as the DRM encryption algorithm). Since the video stream data during target practice training is confidential, to prevent interception during transmission, this invention uses streaming encryption technology to directly encrypt the video stream data stored in each storage block on the disk. When playing the video stream data, the player first uses the corresponding decryption algorithm to decrypt the received video stream data before playback, thus achieving a high degree of confidentiality.

[0059] In addition, in order to further ensure data security, this invention also uses the base64 encoding algorithm to encode the video stream data. When playing the video stream data, the player then uses the corresponding decoding algorithm to decode the video stream data.

[0060] In Embodiment 1 of the present invention, the custom file system for storing video stream data in the custom file system specifically includes:

[0061] In the Linux operating system, each disk of the file storage server is assigned an independent and unique disk index file (i.e., disk index). The disk index file contains the disk's unique serial number and the disk's slot information. In practice, a new disk index file is generated when a disk or slot is replaced.

[0062] Setting the disk to store and retrieve files in fixed-size blocks, meaning that both disk storage and retrieval are done in fixed-size units, can greatly improve data storage efficiency. Since the files being stored are large video files, disk storage is not wasted. For example, in practice, the disk could be set to store and retrieve files in 2MB blocks.

[0063] After the video stream data is obtained from the disk in pass-through mode, the video stream data is divided into storage blocks of fixed size to obtain several small video stream files. At the same time, the offset and size of each small video stream file are calculated based on its position from the file header, and each small video stream file generates a unique offset index file based on its own offset. When accessing the video stream data, sequential reading and writing are performed based on the offset and size of each small video stream file.

[0064] This invention employs a unique disk index file for each disk and a unique offset index file for each video stream file based on its own offset. This allows for accurate and rapid location of the specific storage location and disk information of each video stream file through these two layers of index files, achieving not only security but also significantly improving read and write access efficiency.

[0065] In Embodiment 1 of the present invention, step S2, which involves directly pushing the acquired video stream data to the streaming media server, specifically means: directly pushing the acquired video stream data to the streaming media server via the RTMP protocol. In this invention, after the high-definition camera acquires the video stream data, the RTMP protocol is used to directly push the video stream data to the streaming media server. Since the RTMP push protocol has a relatively small latency, typically less than 100ms, it can better guarantee real-time performance.

[0066] In Embodiment 1 of the present invention, step S2, specifically includes performing frame extraction processing on the video stream data using a streaming media server, including:

[0067] During the training phase of the neural network model, when a target object is detected in the video stream data, the streaming media server acquires images of all frames in the video stream data according to the frame rate. For example, if the frame rate is 30, then 30 frames can be acquired from the video stream data per second. Images without target objects are filtered out, and the remaining images with target objects are used as training materials. Since the video stream data collected before the shooting begins is useless, this invention adopts the method of starting frame extraction only when a target object is detected in the video stream data, which can improve the frame extraction efficiency. At the same time, there may be invalid images (such as no bullets hitting) in the images extracted according to the frame rate. This invention adopts the method of filtering out invalid images directly at the front end, which can greatly improve the efficiency of subsequent modeling.

[0068] In the recognition stage after the neural network model is finalized, when a target object is detected in the video stream data, frames are extracted from the video stream data at preset time intervals, and the number of bullets is determined based on the difference between the target object and the preceding and following frames. In practice, if frames are extracted continuously based on the frame rate, several consecutive images may extract the same bullet, and extracting the correct bullet in every frame also results in a loss of efficiency. Therefore, in the bullet recognition stage, this invention uses preset time intervals to extract frames from the video stream data, improving efficiency by reducing the extraction frequency, while the number of bullets can be determined based on the difference between the preceding and following frames.

[0069] Meanwhile, if the preset time interval is too large, some bullets may not be drawn. Therefore, in order to improve efficiency while ensuring that all bullets can be drawn, the preset time interval in this invention is in the range of 90ms to 110ms; preferably, the preset time interval is 100ms.

[0070] In Embodiment 1 of the present invention, step S3, which involves identifying target objects in the frame-by-frame image using a trained neural network model, specifically includes: creating a neural network model using a neural network algorithm (e.g., the YOLO neural network algorithm); wherein the backbone feature extraction network of the neural network model can be a DarkNet53 network or a CSPDarkNet53 network.

[0071] A large number of images containing target objects are obtained from the training materials to form a dataset. The target objects in each image in the dataset are manually labeled, and the created neural network model is trained using these manually labeled images. This trained neural network model can then simulate human cognition to determine if a new bullet has been fired and programmatically calculate the hit location based on the center point of the identified image markers. Of course, in practical implementation, after the neural network model is trained, it still needs to be optimized (e.g., adjusting video memory and graphics card parameters) and verified to ensure the reliability of the final neural network model.

[0072] Since the number of images selected during the training phase directly affects the accuracy of subsequent recognition by the trained neural network model, and too many images will affect training efficiency, this invention, in its specific implementation, requires obtaining 2800-3200 images with target objects from the training materials as a dataset. Each image's target object is meticulously labeled, including information such as whether a bullet hit and the location of the hit. The more accurate the labeling, the more accurate the final trained model. Actual testing shows that, under the premise of clear images and accurate labeling, using 2800-3200 images with target objects as a dataset for training achieves a neural network model accuracy of 99.99% in the actual testing phase. This helps improve training efficiency while ensuring model accuracy. Preferably, 3000 images with target objects are obtained from the training materials as the dataset.

[0073] During the recognition phase, extracted image frames are input into a trained neural network model to automatically identify the target object. Since the neural network model has already been trained in the early stages, it can automatically make judgments based on machine deep learning during actual use, without requiring any manual intervention on the image, thus effectively ensuring efficiency and real-time performance.

[0074] In its specific implementation, after the neural network model identifies the location of the target object, it calculates and analyzes the bullet's coordinates and the distance between the bullet's firing point and the center point based on the angle and height of the high-definition camera. Based on the ground coordinates of the bullet's impact point on the logical map (i.e., within the image coordinate system), the logical position L of the actual distance from the high-definition camera's ground coordinates is calculated according to the scale. The height of the high-definition camera is H. Assuming the high-definition camera's tilt angle is θ, the actual coordinate point's distance from the high-definition camera's ground coordinates is calculated as L*sinθ within the ground coordinate system.

[0075] In a specific implementation of this invention, bullets can be directly selected as the target object. In this way, when training the neural network model, bullets in each image in the dataset need to be manually labeled, and the created neural network model is trained using the manually labeled images. In the recognition stage, the extracted images are input into the trained neural network model, and the trained neural network model can automatically identify bullets.

[0076] This invention can also select the smoke emitted during bullet firing as the target object. Considering that the smoke won't dissipate quickly after a single shot, the smoke regions of consecutive frames can be used for identification. That is, if the smoke region of the previous frame is a sub-region of the next frame, it's determined to be the same bullet. Thus, the change in smoke regions between consecutive frames allows for a direct determination of whether a new bullet has been fired; and the center of the smoke region indicates the bullet's position. Specifically, a two-dimensional matrix can be used, with one dimension representing the previous frame and the other representing the next. If the previous frame is a sub-region of the next frame, it's recorded as 1; otherwise, it's recorded as 0. When training the neural network model, the smoke in each image of the dataset needs to be manually labeled, and the manually labeled images are used to train the created neural network model. During the recognition phase, extracted frames are input into the trained neural network model, which can automatically identify the smoke, thereby achieving automatic bullet identification.

[0077] In Embodiment 1 of the present invention, step S4, specifically the real-time visualization of the target shooting results, includes displaying the target shooting results using scrolling subtitles and voice broadcasting. In practice, the voice broadcasting can utilize a mature voice module based on machine learning algorithms. The model is trained by inputting a large amount of the user's own voice, and subsequently, the model can support broadcasting text using the user's own voice.

[0078] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0079] Example 2

[0080] This embodiment provides a visual target positioning device, such as Figure 3 As shown, the target positioning device includes a data acquisition and storage module, a streaming and frame extraction module, an object recognition module, and a visualization display module;

[0081] The acquisition and storage module is used to acquire and store video stream data during target shooting training.

[0082] The push-stream frame extraction module is used to directly push the acquired video stream data to the streaming media server, and use the streaming media server to perform frame extraction processing on the video stream data.

[0083] The object recognition module is used to identify target objects in the framed images using a trained neural network model, and generate shooting results based on the position of the identified target objects. The shooting results include information such as whether the target was hit and the position of the hit location from the center of the bullseye.

[0084] The visualization module is used to display the target shooting results in a visual manner in real time.

[0085] This invention employs a method where, after acquiring video stream data during target practice, the video stream data is directly pushed to a streaming media server. The streaming media server then performs frame extraction on the video stream data, and a trained neural network model automatically identifies the extracted images without requiring additional image processing such as grayscale. The entire data processing latency is less than 1 second, resulting in minimal delay. Compared to traditional manual target reporting methods, aside from the initial manual annotation required during training, the entire subsequent recognition process requires no manual intervention, thus significantly reducing workload, improving reporting efficiency, and ensuring safety. Compared to existing methods that require preprocessing of video stream data such as grayscale, this invention effectively reduces latency and ensures real-time data display.

[0086] In a second embodiment of the present invention, the acquisition and storage module specifically includes the following steps: Acquiring and storing video stream data during target practice training.

[0087] High-definition cameras are used to collect video stream data during target practice training. The high-definition cameras are H265 / H264 high-definition cameras, and the image resolution needs to be high enough to clearly see the position of bullets or the base of smoke within a range of 300m. This can ensure the quality of the collected images and thus improve the accuracy of image recognition. In specific implementations, for example, the high-definition camera can be a high-speed infrared PTZ camera. This high-speed infrared PTZ camera uses a CMOS sensor of 1 / 1.8 inches or larger, with a pixel count of no less than 8 million (2688*1520), a zoom of no less than 22x, a focal length range of no less than 6.5-140mm, a frame rate of no less than 60 frames / s, and supports motion detection in at least 18*22 areas simultaneously. It can identify targets at least 500m away from the camera through infrared illumination. Alternatively, the high-definition camera can be an intelligent surveillance PTZ camera with a pixel count of 4 million or larger, an optical zoom of no less than 33x, a focal length range of no less than 6-198mm, a power zoom, and a TVL of no less than 1500 (resolution set to 2688×1520, frame rate set to 30fps, bit rate set to 2Mbps, RJ45 output).

[0088] This invention uses a custom file system to store video stream data instead of a general file system. Only personnel familiar with system programs and commands can access the required content from the custom file system. Others cannot see file attributes and other related information under the Linux file system, thus providing excellent confidentiality.

[0089] The stored video stream data is encrypted using streaming encryption technology (such as the DRM encryption algorithm). Since the video stream data during target practice training is confidential, to prevent interception during transmission, this invention uses streaming encryption technology to directly encrypt the video stream data stored in each storage block on the disk. When playing the video stream data, the player first uses the corresponding decryption algorithm to decrypt the received video stream data before playback, thus achieving a high degree of confidentiality.

[0090] In addition, in order to further ensure data security, this invention also uses the base64 encoding algorithm to encode the video stream data. When playing the video stream data, the player then uses the corresponding decoding algorithm to decode the video stream data.

[0091] In Embodiment 2 of the present invention, the custom file system for storing video stream data in the custom file system specifically includes:

[0092] In the Linux operating system, each disk of the file storage server is assigned an independent and unique disk index file (i.e., disk index). The disk index file contains the disk's unique serial number and the disk's slot information. In practice, a new disk index file is generated when a disk or slot is replaced.

[0093] Setting the disk to store and retrieve files in fixed-size blocks, meaning that both disk storage and retrieval are done in fixed-size units, can greatly improve data storage efficiency. Since the files being stored are large video files, disk storage is not wasted. For example, in practice, the disk could be set to store and retrieve files in 2MB blocks.

[0094] After the video stream data is obtained from the disk in pass-through mode, the video stream data is divided into storage blocks of fixed size to obtain several small video stream files. At the same time, the offset and size of each small video stream file are calculated based on its position from the file header, and each small video stream file generates a unique offset index file based on its own offset. When accessing the video stream data, sequential reading and writing are performed based on the offset and size of each small video stream file.

[0095] This invention employs a unique disk index file for each disk and a unique offset index file for each video stream file based on its own offset. This allows for accurate and rapid location of the specific storage location and disk information of each video stream file through these two layers of index files, achieving not only security but also significantly improving read and write access efficiency.

[0096] In Embodiment 2 of the present invention, in the push-stream frame extraction module, directly pushing the acquired video stream data to the streaming media server specifically means: directly pushing the acquired video stream data to the streaming media server via the RTMP protocol. In this invention, after the high-definition camera acquires the video stream data, the RTMP protocol is used to directly push the video stream data to the streaming media server. Since the RTMP push-stream protocol has a relatively small latency, typically less than 100ms, it can better guarantee real-time performance.

[0097] In a second embodiment of the present invention, the frame extraction process of the video stream data using the streaming media server in the streaming frame extraction module specifically includes:

[0098] During the training phase of the neural network model, when a target object is detected in the video stream data, the streaming media server acquires images of all frames in the video stream data according to the frame rate. For example, if the frame rate is 30, then 30 frames can be acquired from the video stream data per second. Images without target objects are filtered out, and the remaining images with target objects are used as training materials. Since the video stream data collected before the shooting begins is useless, this invention adopts the method of starting frame extraction only when a target object is detected in the video stream data, which can improve the frame extraction efficiency. At the same time, there may be invalid images (such as no bullets hitting) in the images extracted according to the frame rate. This invention adopts the method of filtering out invalid images directly at the front end, which can greatly improve the efficiency of subsequent modeling.

[0099] In the recognition stage after the neural network model is finalized, when a target object is detected in the video stream data, frames are extracted from the video stream data at preset time intervals, and the number of bullets is determined based on the difference between the target object and the preceding and following frames. In practice, if frames are extracted continuously based on the frame rate, several consecutive images may extract the same bullet, and extracting the correct bullet in every frame also results in a loss of efficiency. Therefore, in the bullet recognition stage, this invention uses preset time intervals to extract frames from the video stream data, improving efficiency by reducing the extraction frequency, while the number of bullets can be determined based on the difference between the preceding and following frames.

[0100] Meanwhile, if the preset time interval is too large, some bullets may not be drawn. Therefore, in order to improve efficiency while ensuring that all bullets can be drawn, the preset time interval in this invention is in the range of 90ms to 110ms; preferably, the preset time interval is 100ms.

[0101] In the second embodiment of the present invention, in the object recognition module, the step of performing target object recognition on the frame-by-frame image using a trained neural network model specifically includes: creating a neural network model using a neural network algorithm (e.g., the YOLO neural network algorithm); wherein, the backbone feature extraction network of the neural network model can be a DarkNet53 network or a CSPDarkNet53 network;

[0102] A large number of images containing target objects are obtained from the training materials to form a dataset. The target objects in each image in the dataset are manually labeled, and the created neural network model is trained using these manually labeled images. This trained neural network model can then simulate human cognition to determine if a new bullet has been fired and programmatically calculate the hit location based on the center point of the identified image markers. Of course, in practical implementation, after the neural network model is trained, it still needs to be optimized (e.g., adjusting video memory and graphics card parameters) and verified to ensure the reliability of the final neural network model.

[0103] Since the number of images selected during the training phase directly affects the accuracy of subsequent recognition by the trained neural network model, and too many images will affect training efficiency, this invention, in its specific implementation, requires obtaining 2800-3200 images with target objects from the training materials as a dataset. Each image's target object is meticulously labeled, including information such as whether a bullet hit and the location of the hit. The more accurate the labeling, the more accurate the final trained model. Actual testing shows that, under the premise of clear images and accurate labeling, using 2800-3200 images with target objects as a dataset for training achieves a neural network model accuracy of 99.99% in the actual testing phase. This helps improve training efficiency while ensuring model accuracy. Preferably, 3000 images with target objects are obtained from the training materials as the dataset.

[0104] During the recognition phase, extracted image frames are input into a trained neural network model to automatically identify the target object. Since the neural network model has already been trained in the early stages, it can automatically make judgments based on machine deep learning during actual use, without requiring any manual intervention on the image, thus effectively ensuring efficiency and real-time performance.

[0105] In its specific implementation, after the neural network model identifies the location of the target object, it calculates and analyzes the bullet's coordinates and the distance between the bullet's firing point and the center point based on the angle and height of the high-definition camera. Based on the ground coordinates of the bullet's impact point on the logical map (i.e., within the image coordinate system), the logical position L of the actual distance from the high-definition camera's ground coordinates is calculated according to the scale. The height of the high-definition camera is H. Assuming the high-definition camera's tilt angle is θ, the actual coordinate point's distance from the high-definition camera's ground coordinates is calculated as L*sinθ within the ground coordinate system.

[0106] In a specific implementation of this invention, bullets can be directly selected as the target object. In this way, when training the neural network model, bullets in each image in the dataset need to be manually labeled, and the created neural network model is trained using the manually labeled images. In the recognition stage, the extracted images are input into the trained neural network model, and the trained neural network model can automatically identify bullets.

[0107] This invention can also select the smoke emitted during bullet firing as the target object. Considering that the smoke won't dissipate quickly after a single shot, the smoke regions of consecutive frames can be used for identification. That is, if the smoke region of the previous frame is a sub-region of the next frame, it's determined to be the same bullet. Thus, the change in smoke regions between consecutive frames allows for a direct determination of whether a new bullet has been fired; and the center of the smoke region indicates the bullet's position. Specifically, a two-dimensional matrix can be used, with one dimension representing the previous frame and the other representing the next. If the previous frame is a sub-region of the next frame, it's recorded as 1; otherwise, it's recorded as 0. When training the neural network model, the smoke in each image of the dataset needs to be manually labeled, and the manually labeled images are used to train the created neural network model. During the recognition phase, extracted frames are input into the trained neural network model, which can automatically identify the smoke, thereby achieving automatic bullet identification.

[0108] In Embodiment 2 of the present invention, the real-time visualization of the target shooting results in the visualization display module specifically includes: displaying the target shooting results through scrolling subtitles and voice broadcasting. In specific implementation, the voice broadcasting can employ a relatively mature voice module based on machine learning algorithms. The model is trained by inputting a large amount of the user's own voice in the early stages, and subsequently, the model can support broadcasting using the user's own voice to read text.

[0109] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0110] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to Embodiment 1, as detailed in Embodiment 3.

[0111] Example 3

[0112] This embodiment provides an electronic device, such as... Figure 4As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement any of the embodiments in Example 1.

[0113] Since the electronic device described in this embodiment is the device used to implement the method in Embodiment 1 of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in Embodiment 1 of this application. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection of this application.

[0114] Based on the same inventive concept, this application provides a storage medium corresponding to Embodiment 1, as detailed in Embodiment 4.

[0115] Example 4

[0116] This embodiment provides a computer-readable storage medium, such as... Figure 5 As shown, a computer program is stored thereon, which, when executed by a processor, can implement any of the embodiments in Example 1.

[0117] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method of visualizing targeting positioning, characterized by: The targeting positioning method comprises: The video stream data in the process of the targeting shooting training is collected and stored, specifically including: setting an independent and unique disk index file for each disk of a file storage server, the disk index file containing a unique serial number of the disk and slot information of the disk, a new disk index file being generated when the disk or the slot is replaced; setting the disk to store and read files in a fixed size of a storage block, specifically: setting the disk to store and read files in a fixed size of 2M; after the disk obtains the video stream data through a pass-through mode, dividing the video stream data into a fixed size of a storage block to obtain a plurality of video stream small files; simultaneously calculating the offset and size of each video stream small file according to the position from the file header, and each video stream small file generating a unique offset index file according to the offset thereof; when the video stream data is accessed, sequentially reading and writing according to the offset and size of each video stream small file, and positioning to the specific storage position and disk information of each video stream small file through two layers of index files; The collected video stream data is directly pushed to the streaming media server, and the streaming media server is used to perform frame extraction on the video stream data, specifically including: In the training phase of the neural network model, when a target object appears in the video stream data, the streaming media server obtains images of all frames in the video stream data according to the frame rate, and filters out images without the target object, and the remaining images with the target object are all used as training basic materials; In the recognition phase of the trained neural network model, when a target object appears in the video stream data, the video stream data is extracted at a preset time interval, and the number of bullets is determined according to the difference between the frames before and after the target object; The trained neural network model is used to recognize the target object in the extracted images, and the targeting shooting result is generated according to the position of the recognized target object. After the neural network model recognizes the position of the target object, the coordinates of the bullet and the distance between the bullet shooting point and the center point are calculated according to the angle and height of the high-definition camera; the smoke generated when the bullet is shot is selected as the target object, and a two-dimensional matrix is used, one dimension of which is used to express the previous frame of the bullet, and the other dimension is used to express the next frame of the bullet, if the previous frame is a sub-region of the next frame, then 1 is recorded, otherwise 0 is recorded; The targeting shooting result is displayed in a visual manner in real time.

2. The targeting localization method of claim 1, wherein: The video stream data in the process of the targeting shooting training is collected and stored, specifically including: The video stream data in the process of the targeting shooting training is collected by using a high-definition camera; The video stream data is stored in a self-customized file system; The stored video stream data is encrypted by using a stream encryption technology.

3. The targeting positioning method of claim 1, wherein: The collected video stream data is directly pushed to the streaming media server by using the RTMP protocol.

4. The targeting method of claim 1, wherein: The trained neural network model is used to recognize the target object in the extracted images, specifically including: A neural network model is created by using a neural network algorithm; A large number of images with target objects are obtained from training base materials as a data set, each target object in each image in the data set is manually labeled, and the created neural network model is trained by using the manually labeled images, so as to obtain a trained neural network model; In the identification stage, the frame extracted images are input into the trained neural network model, so as to automatically identify the target objects by using the trained neural network model.

5. The targeting localization method of claim 1, wherein: The shooting result is displayed in a visual manner, specifically including displaying the shooting result in a subtitle scrolling and voice broadcasting manner.

6. A visual targeting positioning device characterized by: The shooting positioning device includes a collection and storage module, a push stream frame extraction module, an object identification module and a visual display module, and is used to realize the method of claim 1; The collection and storage module is used to collect and store video stream data in the shooting training process; The push stream frame extraction module is used to push the collected video stream data to a streaming media server directly, and perform frame extraction on the video stream data by using the streaming media server, specifically including: In the training stage of the neural network model, when the target object appears in the video stream data, the streaming media server acquires all frame images in the video stream data according to the frame rate, and filters out the images without the target object, and the remaining images with the target object are all used as training base materials; In the identification stage after the neural network model is shaped, when the target object appears in the video stream data, the video stream data is frame extracted at a preset time interval, and the number of bullets is determined according to the difference between the front and back frames of the target object; The object identification module is used to identify the target object in the frame extracted image by using the trained neural network model, and generate the shooting result according to the position of the identified target object; The visual display module is used to display the shooting result in a visual manner.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the method of any one of claims 1 to 5.

Images