Target detection tracking method and device, electronic equipment and storage medium

Abstract

The invention provides a target detection tracking method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a wide-angle field video stream of a target area in real time through one or more spherical cameras; decomposing the wide-angle field video stream into a plurality of continuous frame images, performing moving target detection on each image, and identifying moving targets of different sizes; associating and tracking the same moving target in the continuous frames of images, and updating the track information of the moving target; overlapping the detected moving target and the track information corresponding to the moving target on the wide-angle view field video stream, and displaying the overlapped wide-angle view field video stream on the user interface in real time. The number of cameras can be reduced by using the spherical camera, so that the monitoring cost is reduced, the moving target detection is carried out on each image, and the monitoring efficiency is improved. Moving targets of different sizes can be recognized, objects in long-range images or objects of small sizes can be accurately recognized, and safety is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and tracking, and in particular to a target detection and tracking method, device, electronic equipment and storage medium. Background Art

[0002] Security area monitoring plays a vital role in key scenarios such as warehouses, high-voltage substations, and aprons. In these specific environments, continuous and efficient monitoring of intruders is an indispensable part of building a safety protection network. Traditional security area monitoring uses a monocular camera to obtain security area images, and uses a target recognition algorithm to identify whether there are intruders in the acquired security area images. However, in the prior art, due to the limited field of view of the monocular camera, multiple cameras need to be configured, which increases the monitoring cost. In addition, due to the low target recognition accuracy of the traditional target recognition algorithm, only larger objects can be recognized. Objects in distant images or smaller objects cannot be accurately recognized, which poses a risk of safety accidents. Summary of the invention

[0003] The present invention provides a target detection and tracking method, device, electronic device and storage medium, which are used to solve the defects of traditional target detection and tracking methods, such as high monitoring cost, inability to accurately identify objects in distant images or objects of small size, and potential safety accidents.

[0004] The present invention provides a target detection and tracking method, comprising: Acquire a wide-angle field-of-view video stream of a target area in real time through one or more spherical cameras; Decomposing the wide-angle video stream into a plurality of continuous frame images, performing moving target detection on each image, and identifying moving targets of different sizes; Associating and tracking the same moving target in consecutive frame images, and updating the trajectory information of the moving target; The detected moving target and its corresponding trajectory information are superimposed on the wide-angle field of view video stream, and the superimposed wide-angle field of view video stream is displayed in real time on the user interface.

[0005] According to the target detection and tracking method provided by the present invention, the moving target detection for each image includes: Decoding the video stream by using the development toolkit of the spherical camera, and retrieving YUV format image data of multiple consecutive frames; Use FFMPEG tool to convert the YUV format image data into BGR format image data; Perform moving target detection on BGR format image data.

[0006] According to the target detection and tracking method provided by the present invention, the control method of the one or more spherical cameras includes: Through the http interface connected to the spherical camera, the corresponding spherical camera is turned on or off according to the position and parameters of each spherical camera.

[0007] According to the target detection and tracking method provided by the present invention, after associating and tracking the same moving target in the continuous frame images, the method further includes: Obtaining the current position, moving direction and speed of the moving target in the wide-angle field of view of the spherical camera; Based on the current position, movement direction and speed of the moving target in the wide-angle field of view of the spherical camera, the possible future position of the moving target is estimated by using the Kalman algorithm; A control instruction is generated according to the possible future position of the moving target, so as to adjust the direction and angle of the spherical camera according to the control instruction.

[0008] According to the target detection and tracking method provided by the present invention, the control instruction includes at least one of a rotation angle, a pitch angle and a zoom, and adjusting the direction and angle of the spherical camera according to the control instruction includes: Adjusting the direction and angle of the spherical camera in combination with the physical limitations of the spherical camera, shooting requirements and the control instructions; When the control instruction is a rotation angle, determine whether the angle to be rotated in the control instruction exceeds the maximum horizontal angle range of the spherical camera after being superimposed on the current horizontal angle; if not, rotate according to the control instruction; otherwise, reset the spherical camera to a preset position; When the control instruction is a pitch angle, determine whether the pitch angle to be pitched in the control instruction exceeds the maximum pitch angle range of the spherical camera after being superimposed with the current pitch angle; if not, make adjustments according to the control instruction; otherwise, reset the spherical camera to a preset position; When the control instruction is zoom, it is determined whether the zoom parameter in the control instruction exceeds the zoom speed and resolution limit of the spherical camera. If not, adjustment is made according to the control instruction; otherwise, the focus of the spherical camera is not adjusted.

[0009] According to the target detection and tracking method provided by the present invention, the moving target detection is performed on each image to identify moving targets of different sizes, including: Input multiple continuous frame images into the target detection model based on YOLOv8 and output moving targets of different sizes; The YOLOv8-based target detection model is trained based on multiple historical objects that have intruded into safe areas.

[0010] According to the target detection and tracking method provided by the present invention, the training method of the target detection model based on YOLOv8 includes: The images in the training data whose target size is larger than a preset threshold are directly input into the YOLOv8-based target detection model, and the large target prediction result is output; Calculate a first loss between the large target prediction result and the large target calibration result, optimize the parameters of the YOLOv8-based target detection model according to the first loss, and output a large target training completion model; Cropping the image whose target size is larger than a preset size threshold to enlarge the proportion of the target in the cropped image; Filling the image in which the proportion of the target in the cropped image is greater than a preset proportion threshold, inputting the filled image into the large target training completion model, and outputting the small target prediction result; Calculate the second loss between the small target prediction result and the small target calibration result, optimize the parameters of the YOLOv8-based target detection model according to the second loss, output the small target training completion model, and use the small target training completion model as the trained YOLOv8-based target detection model.

[0011] The present invention also provides a target detection and tracking device, comprising: An acquisition module, used for acquiring a wide-angle field-of-view video stream of a target area in real time through one or more spherical cameras; A recognition module, used to decompose the wide-angle field of view video stream into multiple continuous frame images, perform moving target detection on each image, and identify moving targets of different sizes; An updating module, used for associating and tracking the same moving target existing in consecutive frame images, and updating the trajectory information of the moving target; The display module is used to superimpose the detected moving target and its corresponding trajectory information on the wide-angle field of view video stream, and display the superimposed wide-angle field of view video stream in real time on the user interface.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the target detection and tracking method as described in any one of the above items is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the target detection and tracking methods described above.

[0014] The target detection and tracking method, device, electronic device and storage medium provided by the present invention obtain a wide-angle field of view video stream of a target area in real time through one or more spherical cameras; decompose the wide-angle field of view video stream into multiple continuous frame images, perform motion target detection on each image, and identify motion targets of different sizes; associate and track the same motion target existing in the continuous frame images, and update the trajectory information of the motion target; superimpose the detected motion target and its corresponding trajectory information on the wide-angle field of view video stream, and display the superimposed wide-angle field of view video stream in real time on a user interface. The present invention uses spherical cameras to reduce the number of cameras, thereby reducing monitoring costs. By performing motion target detection on each image, moving targets of different sizes can be identified. Objects in distant images or objects of smaller sizes can also be accurately identified, thereby improving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is one of the flow charts of the target detection and tracking method provided by an embodiment of the present invention; Figure 2 This is a second flow chart of the target detection and tracking method provided by an embodiment of the present invention; Figure 3 is a functional structure diagram of a target detection and tracking device provided by an embodiment of the present invention; Figure 4 is a functional structure diagram of a target detection and tracking system provided by an embodiment of the present invention; Figure 5 It is a functional structure diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Figure 1 A flow chart of a target detection and tracking method provided by an embodiment of the present invention, such as Figure 1As shown, the target detection and tracking method provided by the embodiment of the present invention includes: Step 101: acquiring a wide-angle field-of-view video stream of a target area in real time through one or more spherical cameras; Step 102: decompose the wide-angle video stream into multiple continuous frame images, perform moving target detection on each image, and identify moving targets of different sizes; Step 103: Associating and tracking the same moving target in consecutive frame images, and updating the trajectory information of the moving target; In the embodiment of the present invention, the Bytetrack target tracking algorithm is used to associate and track targets in consecutive frames. The Bytetrack target tracking algorithm can keep tracking the same target in consecutive frames based on information such as the target's appearance features and motion features. The target's trajectory information, including position, speed, acceleration, etc., is updated to facilitate subsequent analysis and processing.

[0019] Step 104: superimpose the detected moving target and its corresponding trajectory information on the wide-angle field of view video stream, and display the superimposed wide-angle field of view video stream in real time on the user interface.

[0020] In traditional target detection and tracking methods, due to the limited field of view of a monocular camera, multiple cameras need to be configured, which increases the monitoring cost. In addition, due to the low target recognition accuracy of traditional target recognition algorithms, only larger objects can be recognized. Objects in distant images or smaller objects cannot be accurately recognized, posing a risk of safety accidents.

[0021] The target detection and tracking method provided by the embodiment of the present invention obtains a wide-angle field of view video stream of a target area in real time through one or more spherical cameras; decomposes the wide-angle field of view video stream into multiple continuous frame images, performs motion target detection on each image, and identifies motion targets of different sizes; associates and tracks the same motion target existing in the continuous frame images, and updates the trajectory information of the motion target; superimposes the detected motion target and its corresponding trajectory information on the wide-angle field of view video stream, and displays the superimposed wide-angle field of view video stream in real time on a user interface. The present invention uses spherical cameras to reduce the number of cameras, thereby reducing monitoring costs. By performing motion target detection on each image, moving targets of different sizes can be identified. Objects in distant images or objects of smaller sizes can also be accurately identified, thereby improving safety.

[0022] Based on any of the above embodiments, the step of performing moving target detection on each image includes: Step 201: Decode the video stream through the development toolkit of the spherical camera to retrieve YUV format image data of multiple consecutive frames; Step 202: Use FFMPEG tool to convert the YUV format image data into BGR format image data; Step 203: Perform moving target detection on the image data in BGR format.

[0023] Based on any of the above embodiments, the control method of the one or more spherical cameras includes: Through the http interface connected to the spherical camera, the corresponding spherical camera is turned on or off according to the position and parameters of each spherical camera.

[0024] In the embodiment of the present invention, the system online service provides two external http interfaces, namely, to enable or disable algorithm processing and to query whether the camera is currently enabled or disabled.

[0025] Based on any of the above embodiments, after associating and tracking the same moving target in the consecutive frame images, the method further includes: Step 301, obtaining the current position, moving direction and speed of the moving target in the wide-angle field of view of the spherical camera; Step 302: Estimate the possible future position of the moving target through the Kalman algorithm based on the current position, moving direction and speed of the moving target in the wide-angle field of view of the spherical camera; Step 303: Generate a control instruction according to the possible future position of the moving target, so as to adjust the direction and angle of the spherical camera according to the control instruction.

[0026] In the embodiment of the present invention, the control instruction includes at least one of a rotation angle, a pitch angle, and a zoom, and adjusting the direction and angle of the spherical camera according to the control instruction includes: Adjusting the direction and angle of the spherical camera in combination with the physical limitations of the spherical camera, shooting requirements and the control instructions; In the embodiment of the present invention, the shooting requirements include but are not limited to keeping the target in the center of the picture, maintaining an appropriate distance, etc.

[0027] When the control instruction is a rotation angle, determine whether the angle to be rotated in the control instruction exceeds the maximum horizontal angle range of the spherical camera after being superimposed on the current horizontal angle; if not, rotate according to the control instruction; otherwise, reset the spherical camera to a preset position; When the control instruction is a pitch angle, determine whether the pitch angle to be pitched in the control instruction exceeds the maximum pitch angle range of the spherical camera after being superimposed with the current pitch angle; if not, make adjustments according to the control instruction; otherwise, reset the spherical camera to a preset position; When the control instruction is zoom, it is determined whether the zoom parameter in the control instruction exceeds the zoom speed and resolution limit of the spherical camera. If not, adjustment is made according to the control instruction; otherwise, the focus of the spherical camera is not adjusted.

[0028] According to the target detection and tracking method provided by the present invention, the moving target detection is performed on each image to identify moving targets of different sizes, including: Input multiple continuous frame images into the target detection model based on YOLOv8 and output moving targets of different sizes; The YOLOv8-based target detection model is trained based on multiple historical objects that have intruded into safe areas.

[0029] Based on any of the above embodiments, the training method of the target detection model based on YOLOv8 includes: The images in the training data whose target size is larger than a preset threshold are directly input into the YOLOv8-based target detection model, and the large target prediction result is output; Calculate a first loss between the large target prediction result and the large target calibration result, optimize the parameters of the YOLOv8-based target detection model according to the first loss, and output a large target training completion model; Cropping the image whose target size is larger than a preset size threshold to enlarge the proportion of the target in the cropped image; Filling the image in which the proportion of the target in the cropped image is greater than a preset proportion threshold, inputting the filled image into the large target training completion model, and outputting the small target prediction result; Calculate the second loss between the small target prediction result and the small target calibration result, optimize the parameters of the YOLOv8-based target detection model according to the second loss, output the small target training completion model, and use the small target training completion model as the trained YOLOv8-based target detection model.

[0030] In an embodiment of the present invention, training the model based on small target training data can improve the model's recognition accuracy for distant targets and smaller objects, and can be applied to larger plant monitoring and management scenarios.

[0031] In the embodiment of the present invention, alarm processing is also included. When an intruder target is detected, the alarm result needs to be managed, including obtaining the alarm process such as the time, location, target type and kinematic characteristics and direction of the moving target, publishing the alarm result and recording the alarm video.

[0032] like Figure 2As shown, the target detection and tracking method provided by the embodiment of the present invention also includes: obtaining the original data of the video stream, decoding the original data of the video stream, managing the camera, and judging whether the camera is turned on. In the embodiment of the present invention, judging whether the camera is turned on can be determined by the camera parameter value, or by obtaining the camera video stream; if the camera is turned on and online, detecting and tracking the target; judging whether the moving target breaks into the target area, if so, controlling the pan-tilt tracking, and resetting the camera to the preset position if the moving target exceeds the camera tracking boundary. The target area is set by the user according to the usage scenario.

[0033] The target detection and tracking method provided by the embodiment of the present invention can efficiently control the spherical camera, and utilize the omnidirectional field of view of the spherical camera to perform uninterrupted and accurate real-time monitoring of the target scene. It not only supports the access of multiple cameras, but also allows users to flexibly add or delete cameras online, greatly enhancing the flexibility and scalability of the monitoring system. In particular, when a person or vehicle is detected to break into a preset sensitive area, the tracking algorithm is automatically activated immediately, and the instant alarm mechanism is triggered to ensure that any abnormal behavior can be quickly captured and responded to. At the same time, in addition, it also has a complete alarm information management function, which can comprehensively record all alarm events of the alarm camera within a specified time period, including detailed information such as time, location, and target type, thereby solving the problem of personnel misjudgment caused by limited monitoring range or incomplete information recording in traditional monitoring systems to a certain extent, significantly reducing the risk of safety accidents, improving the level of safety protection, and optimizing the monitoring management process.

[0034] The target detection and tracking method provided by the embodiment of the present invention can be applied to scenarios such as aviation manufacturing, building construction, shipping ports, energy supply, emergency rescue, etc.

[0035] The target detection and tracking device provided by the present invention is described below. The target detection and tracking device described below and the target detection and tracking method described above can be referred to each other.

[0036] Figure 3 A schematic diagram of the structure of a target detection and tracking device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the target detection and tracking device provided by the embodiment of the present invention includes: An acquisition module 301 is used to acquire a wide-angle field-of-view video stream of a target area in real time through one or more spherical cameras; The recognition module 302 is used to decompose the wide-angle field of view video stream into multiple continuous frame images, perform moving target detection on each image, and identify moving targets of different sizes; An updating module 303 is used to associate and track the same moving target in consecutive frame images and update the trajectory information of the moving target; The display module 304 is used to superimpose the detected moving target and its corresponding trajectory information on the wide-angle field of view video stream, and display the superimposed wide-angle field of view video stream in real time on the user interface.

[0037] The target detection and tracking device provided by the embodiment of the present invention obtains a wide-angle field of view video stream of a target area in real time through one or more spherical cameras; decomposes the wide-angle field of view video stream into multiple continuous frame images, performs motion target detection on each image, and identifies moving targets of different sizes; associates and tracks the same moving target existing in the continuous frame images, and updates the trajectory information of the moving target; superimposes the detected moving target and its corresponding trajectory information on the wide-angle field of view video stream, and displays the superimposed wide-angle field of view video stream in real time on a user interface. The present invention uses spherical cameras to reduce the number of cameras, thereby reducing monitoring costs. By performing motion target detection on each image, moving targets of different sizes can be identified. Objects in distant images or objects of smaller sizes can also be accurately identified, thereby improving safety.

[0038] The embodiment of the present invention also provides a target detection and tracking system. Figure 4 As shown, it includes: a ball camera, a GPU server, and a camera NVR device, wherein the GPU server includes a software system, and the software system includes: Camera management module: supports online access of multiple cameras and video stream processing.

[0039] Algorithm management module: including pedestrian and vehicle detection, target tracking, and Kalman filtering.

[0040] Camera gimbal motion control module: Keep the camera facing the target to be tracked through target monitoring and tracking results.

[0041] Alarm module: save and push violation results.

[0042] Front-end and back-end modules: The front-end is mainly responsible for the user's interactive front-end page and the back-end is responsible for user and camera resource management.

[0043] Object storage module: stores alarm results and displays them on the front-end page.

[0044] The embodiment of the present invention is based on multi-channel online camera management, moving target detection and tracking, camera pan / tilt motion control, and target detection and tracking, specifically including: Step 1. When the user turns on a camera through the online service, the system will log in to the camera through the camera network software development kit, obtain the camera's real-time stream decoding and de-framing into image data, and then transmit the image data to the algorithm management module for specific algorithm processing. The detailed steps are as follows: The online service provides two external http interfaces, namely, turning the algorithm on and off and querying the current camera status. To obtain the camera stream data, you first need to remotely connect to the camera and decode the camera's YUV format image data through the camera's development kit, and then use the FFMPEG tool to convert it into BGR format image data and send it to the data bus for subsequent algorithm processing. Turning on and off the camera algorithm includes allocating system resources, obtaining real-time image data from the data bus, loading the algorithm, and turning off the algorithm.

[0045] Step 2. Moving target detection and tracking. In the moving target detection and tracking stage, the main goal is to analyze the image data obtained from the camera in real time, identify the moving target, and track it continuously. The deep learning model YOLOv8 is used to detect the target on the input BGR format image. These models are trained to identify specific targets (such as people, vehicles, etc.) in the image and mark the detected targets, including the location (bounding box), category, and confidence of the target. For targets in consecutive frames, the bytetrack target tracking algorithm is used for association and tracking. These algorithms can keep tracking the same target in consecutive frames based on information such as the target's appearance features and motion features. Update the target's trajectory information, including position, speed, acceleration, etc., for subsequent analysis and processing. The detected target and its tracking results are superimposed on the original image in a visual form (such as bounding boxes, trajectory lines, etc.) and displayed in real time on the user interface. The detection results and tracking data are sent to the data bus or storage system in a structured form for subsequent analysis and alarm use.

[0046] Step 3. Camera gimbal motion control. The purpose of camera gimbal motion control is to adjust the camera's viewing angle according to the position and dynamics of the moving target to achieve the best shooting effect of the target. According to the results of target detection and tracking, analyze the position, movement direction and speed of the target in the current camera field of view. Use Kalman to estimate the possible future position of the target, and combine the target tracking results to determine the adjustment direction and angle of the camera gimbal. According to the target position analysis results, generate corresponding gimbal control instructions, including rotation angle, pitch angle, zoom, etc. The instructions should take into account the physical limitations of the camera (such as rotation range, speed limit, etc.) and shooting requirements (such as keeping the target in the center of the picture, maintaining an appropriate distance, etc.). Send the generated gimbal control instructions to the camera gimbal control system. Receive feedback from the gimbal control system in real time to ensure that the instructions are executed correctly, and adjust the instructions as needed to deal with unexpected situations (such as the target suddenly changes direction).

[0047] Step 4. Alarm processing. When an intruder is detected, the alarm results need to be managed, including obtaining the alarm process, publishing the alarm results, and recording the alarm video. The system can record and upload the tracking screen to the server in real time, providing high-definition and detailed image data for remote monitoring and post-analysis. This function effectively promotes the instant transmission of information, allowing on-site staff to quickly receive alarm signals, make accurate judgments based on real-time images, and take necessary safety measures in a timely manner.

[0048] The present invention uses illegal intrusion into a safe area as a trigger condition for camera tracking, combines deep learning knowledge with pan-tilt motion control methods, and can identify and track targets with high precision, providing a feasible solution for applying artificial intelligence technology in this field to improve monitoring effects.

[0049] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The memory 530 includes a computer program, an operating system and acquired data, and the processor 510 may call the logic instructions in the memory 530 to execute the target detection and tracking method, which includes: one or more spherical cameras acquire a wide-angle field of view video stream of a target area in real time; decompose the wide-angle field of view video stream into multiple continuous frame images, perform motion target detection on each image, and identify motion targets of different sizes; associate and track the same motion target in the continuous frame images, and update the trajectory information of the motion target; superimpose the detected motion target and its corresponding trajectory information on the wide-angle field of view video stream, and display the superimposed wide-angle field of view video stream in real time on the user interface.

[0050] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0051] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the target detection and tracking method provided by the above-mentioned methods, the method comprising: one or more spherical cameras acquiring a wide-angle field-of-view video stream of a target area in real time; decomposing the wide-angle field-of-view video stream into a plurality of continuous frame images, performing motion target detection on each image, and identifying motion targets of different sizes; associating and tracking the same motion target existing in continuous frame images, and updating the trajectory information of the motion target; superimposing the detected motion target and its corresponding trajectory information on the wide-angle field-of-view video stream, and displaying the superimposed wide-angle field-of-view video stream in real time on a user interface.

[0052] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0053] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection and tracking method, characterized in that: include: Acquire a wide-angle field-of-view video stream of a target area in real time through one or more spherical cameras; Decomposing the wide-angle video stream into a plurality of continuous frame images, performing moving target detection on each image, and identifying moving targets of different sizes; Associating and tracking the same moving target in consecutive frame images, and updating the trajectory information of the moving target; The detected moving target and its corresponding trajectory information are superimposed on the wide-angle field of view video stream, and the superimposed wide-angle field of view video stream is displayed in real time on the user interface.

2. The target detection and tracking method according to claim 1, characterized in that: The step of performing moving target detection on each image includes: Decoding the video stream by using the development toolkit of the spherical camera, and retrieving YUV format image data of multiple consecutive frames; Using FFMPEG tool to convert the YUV format image data into BGR format image data; Perform moving target detection on BGR format image data.

3. The target detection and tracking method according to claim 1, characterized in that: The control method of the one or more spherical cameras comprises: Through the http interface connected to the spherical camera, the corresponding spherical camera is turned on or off according to the position and parameters of each spherical camera.

4. The target detection and tracking method according to claim 1, characterized in that: After associating and tracking the same moving target in the continuous frame images, the method further includes: Obtaining the current position, moving direction and speed of the moving target in the wide-angle field of view of the spherical camera; Based on the current position, movement direction and speed of the moving target in the wide-angle field of view of the spherical camera, the possible future position of the moving target is estimated by using the Kalman algorithm; A control instruction is generated according to the possible future position of the moving target, so as to adjust the direction and angle of the spherical camera according to the control instruction.

5. The target detection and tracking method according to claim 4, characterized in that: The control instruction includes at least one of a rotation angle, a pitch angle, and a zoom, and adjusting the direction and angle of the spherical camera according to the control instruction includes: Adjusting the direction and angle of the spherical camera in combination with the physical limitations of the spherical camera, shooting requirements and the control instructions; When the control instruction is a rotation angle, determine whether the angle to be rotated in the control instruction exceeds the maximum horizontal angle range of the spherical camera after being superimposed on the current horizontal angle; if not, rotate according to the control instruction; otherwise, reset the spherical camera to a preset position; When the control instruction is a pitch angle, determine whether the pitch angle to be pitched in the control instruction exceeds the maximum pitch angle range of the spherical camera after being superimposed with the current pitch angle; if not, make adjustments according to the control instruction; otherwise, reset the spherical camera to a preset position; When the control instruction is zoom, it is determined whether the zoom parameter in the control instruction exceeds the zoom speed and resolution limit of the spherical camera. If not, adjustment is made according to the control instruction; otherwise, the focus of the spherical camera is not adjusted.

6. The target detection and tracking method according to claim 1, characterized in that: The moving target detection is performed on each image to identify moving targets of different sizes, including: Input multiple continuous frame images into the target detection model based on YOLOv8 and output moving targets of different sizes; The YOLOv8-based target detection model is trained based on multiple historical objects that have intruded into safe areas.

7. The target detection and tracking method according to claim 6, characterized in that: The training method of the target detection model based on YOLOv8 includes: The images in the training data whose target size is larger than a preset threshold are directly input into the YOLOv8-based target detection model, and the large target prediction result is output; Calculate a first loss between the large target prediction result and the large target calibration result, optimize the parameters of the YOLOv8-based target detection model according to the first loss, and output a large target training completion model; Cropping the image whose target size is larger than a preset size threshold to enlarge the proportion of the target in the cropped image; Filling the image in which the proportion of the target in the cropped image is greater than a preset proportion threshold, inputting the filled image into the large target training completion model, and outputting the small target prediction result; Calculate the second loss between the small target prediction result and the small target calibration result, optimize the parameters of the YOLOv8-based target detection model according to the second loss, output the small target training completion model, and use the small target training completion model as the trained YOLOv8-based target detection model.

8. A target detection and tracking device, characterized in that: include: An acquisition module, used for acquiring a wide-angle field-of-view video stream of a target area in real time through one or more spherical cameras; A recognition module, used to decompose the wide-angle field of view video stream into multiple continuous frame images, perform moving target detection on each image, and identify moving targets of different sizes; An updating module, used for associating and tracking the same moving target existing in consecutive frame images, and updating the trajectory information of the moving target; The display module is used to superimpose the detected moving target and its corresponding trajectory information on the wide-angle field of view video stream, and display the superimposed wide-angle field of view video stream in real time on the user interface.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the target detection and tracking method according to any one of claims 1 to 7 is implemented.

10. A non-transitory readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target detection and tracking method as described in any one of claims 1 to 7 is implemented.

Images