A target detection method, apparatus, electronic device, and storage medium

Abstract

This invention provides a target detection method, apparatus, electronic device, and storage medium. The method includes: acquiring multiple groups of consecutive video frames sent by a terminal, wherein each group of consecutive video frames includes a first video frame and N second video frames; performing target detection on each of the first video frames to obtain first target location information in each of the first video frames, and sending each of the first target location information to the terminal so that the terminal can determine first target interest range information of the second video frames in each group of consecutive video frames based on each of the first target location information and through a target tracking algorithm; and acquiring the first target interest range information fed back by the terminal.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a target detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of computer technology, target detection in video images has been widely applied.

[0003] However, in existing technologies, the computational work of target detection is often performed by terminal devices. However, due to the limited computing power of the terminal, it is often difficult to handle a large number of computational tasks. Therefore, it is often necessary to use cloud platform servers to share the computational tasks. However, the current technology of terminal-cloud collaborative target detection has a high demand for network bandwidth.

[0004] Therefore, how to achieve object detection has become an urgent problem to be solved in the industry. Summary of the Invention

[0005] This invention provides a target detection method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that require high network bandwidth for target detection through edge-cloud collaboration.

[0006] This invention provides a target detection method, comprising:

[0007] Acquire multiple groups of consecutive video frames sent by the terminal, wherein each group of consecutive video frames includes a first video frame and N second video frames;

[0008] Target detection is performed on each of the first video frames to obtain the first target location information in each of the first video frames, and the first target location information is sent to the terminal so that the terminal can determine the first target interest range information of the second video frame in each group of consecutive video frames based on the first target location information and through a target tracking algorithm.

[0009] Obtain the first target interest range information fed back by the terminal.

[0010] According to a target detection method provided by the present invention, after obtaining the first target location information in each target video frame, the method further includes:

[0011] Obtain the first target interest range information of the target second video frame, wherein the target second video frame is the video frame preceding the first video frame;

[0012] Target association analysis is performed on the target interest range information of the second video frame and the first target location information of the first video frame to determine the second target location information in the first video frame.

[0013] According to a target detection method provided by the present invention, after determining the location information of a second target in the first video frame, the method further includes:

[0014] The second target location information is sent to the terminal so that the terminal can determine the second target interest range information in the continuous video frame group where the second video frame of the target is located based on the second target location information and through a target tracking algorithm;

[0015] Obtain the second target interest range information fed back by the terminal.

[0016] According to a target detection method provided by the present invention, after determining the location information of a second target in the first video frame, the method further includes:

[0017] Based on the number of second targets in the second target location, determine the occurrence ratio information of the second targets;

[0018] Based on the second target occurrence ratio information, the length information of the continuous video frame group is determined;

[0019] The length information is sent to the terminal so that the terminal can regroup the original video according to the length information to obtain multiple groups of consecutive video frames after regrouping.

[0020] According to a target detection method provided by the present invention, determining the length information of the consecutive video frame group based on the second target occurrence ratio information includes:

[0021]

[0022] Where N is the original length information of the consecutive video frame group, N max and N min These are the maximum and minimum preset lengths of the frame group, R. th1 and R th2 It is a preset occurrence ratio threshold, N′ is the length information of the continuous video frame group, and R is the occurrence ratio information of the second target.

[0023] The present invention also provides a target detection device, comprising:

[0024] The first acquisition module is used to acquire multiple groups of consecutive video frames sent by the terminal, wherein each group of consecutive video frames includes a first video frame and N second video frames.

[0025] The detection module is used to perform target detection on each of the first video frames, obtain the first target location information in each of the first video frames, and send the first target location information to the terminal so that the terminal can determine the first target interest range information of the second video frame in each group of the consecutive video frames based on the first target location information and through a target tracking algorithm.

[0026] The second acquisition module is used to acquire the first target interest range information fed back by the terminal.

[0027] According to the target detection device provided by the present invention, the device further includes:

[0028] The third acquisition module is used to acquire the first target interest range information of the target second video frame, wherein the target second video frame is the video frame preceding the first video frame.

[0029] The analysis module is used to perform target association analysis on the target interest range information of the second video frame and the first target location information of the first video frame to determine the second target location information in the first video frame.

[0030] According to the target detection device provided by the present invention, the device further includes:

[0031] The sending module is used to send the second target location information to the terminal, so that the terminal can determine the second target interest range information in the continuous video frame group where the second video frame of the target is located based on the second target location information and through a target tracking algorithm;

[0032] The fourth acquisition module is used to acquire the second target interest range information fed back by the terminal.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the target detection methods described above.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the target detection methods described above.

[0035] The present invention provides a target detection method, apparatus, electronic device, and storage medium. After acquiring a continuous group of video frames sent by a terminal, the terminal first performs target detection only on the first video frame to determine the location information of the first target. Then, based on the location information of the first target, the terminal retains only the area in the second video frame where the location information of the first target may exist, that is, only the range of interest information of the first target is retained. During data upload, the terminal discards data in the second video frame other than the range of interest information of the first target. This can significantly reduce the bandwidth required for video data upload while achieving effective target detection. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 The flowchart of the target detection method provided by the present invention;

[0038] Figure 2 This is a schematic diagram of the target detection device provided by the present invention;

[0039] Figure 3 A schematic diagram of the physical structure of an electronic device is provided. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] Figure 1 The flowchart of the target detection method provided by the present invention is as follows: Figure 1 As shown, it includes:

[0042] Step 110: Obtain multiple groups of consecutive video frames sent by the terminal, wherein each group of consecutive video frames includes a first video frame and N second video frames;

[0043] Specifically, the cloud platform acquires multiple sets of continuous video frames sent by the terminal. The cloud platform can specifically refer to a cloud server capable of target detection, and the cloud platform can specifically be equipped with intelligent target detection algorithms to achieve target detection.

[0044] The terminal described in this application can specifically be a terminal with data transmission function, such as a mobile phone, tablet computer, camera, or processor installed in a car, etc.

[0045] The multiple groups of continuous video frames described in this application embodiment are achieved by the terminal adaptively aggregating the input video frame according to a preset length information N, and adaptively dividing the video frame into multiple groups. Each video frame in each group has temporal continuity, that is, each continuous video frame group contains temporally continuous video frames.

[0046] The first video frame in this application is a key video frame selected from a group of consecutive video frames. It can be the first frame in the group of consecutive video frames or any frame selected from the group of consecutive video frames.

[0047] The second video frame described in this application is any other video frame in a group of consecutive video frames besides the first video frame, and there may be one or more of them.

[0048] Step 120: Target detection is performed on each of the first video frames to obtain the first target location information in each of the first video frames, and the first target location information is sent to the terminal so that the terminal can determine the first target interest range information of the second video frame in each group of consecutive video frames based on the first target location information and through a target tracking algorithm.

[0049] Specifically, the cloud platform identifies the region of interest in the first video frame through a target detection neural network algorithm, obtains the location information of the first target, and feeds back the location range information to the terminal device. The target detection neural network algorithm can be a pre-trained target detection neural network algorithm.

[0050] The first target location information described in this application may specifically refer to information about the location of the detected target.

[0051] After the cloud platform detects the location information of the first target in each first video frame, it sends the location information of each first target to the terminal. After the terminal device obtains the location information of the first target, it uses the location as the initial value, calls the target tracking algorithm, and calculates the possible location range of the target in the subsequent second video frames in the same frame group, that is, the first target interest range information.

[0052] In this embodiment, a square region is first selected for each possible target location range. Considering potential errors in the tracking algorithm, the four sides of this square region are extended outward by n pixels. To reduce image artifacts and additional bandwidth requirements caused by high-frequency components within macroblocks in common video / image coding, the sides of this region are further extended outward until they align with coordinates that are multiples of 8. This region boundary extension process is expressed by the formula:

[0053]

[0054]

[0055] In the formula, (x0,y0) and (x1,y1) are the coordinates of the top-left and bottom-right corners of the square region obtained by the tracking algorithm; (x′0,y′0) and (x′1,y′1) are the coordinates of the top-left and bottom-right corners of the extended square region. W and H are the width and height pixels of each frame. The value of n is given by the following formula:

[0056] n = n0 + Δn·v

[0057] In the formula, v represents the target's movement speed in the current frame (unit: pixels / frame). In this invention, n0 = 4 and Δn = 0.5.

[0058] Next, the union of the square regions extended from all targets is calculated. Only the image regions within this union are encoded and uploaded to the cloud device via the network. The cloud device then runs a target detection neural network algorithm to determine the precise location of the target within the image, thus obtaining the first target area of ​​interest information.

[0059] Step 130: Obtain the first target interest range information fed back by the terminal.

[0060] Specifically, in order to effectively reduce the bandwidth required for data transmission between the cloud platform and the terminal, non-interest area image data in the second video frame is discarded during data upload, which greatly reduces the bandwidth required for video data upload. Therefore, the terminal only feeds back image data within the first target interest area information in each video frame group.

[0061] In this embodiment, after obtaining a continuous group of video frames sent by the terminal, target detection is first performed only on the first video frame to determine the first target location information. Then, based on the first target location information, the terminal retains only the area where the first target location information may exist in the second video frame, that is, only the first target interest range information is retained. During data upload, data in the second video frame other than the first target interest range information is discarded. This can significantly reduce the bandwidth required for video data upload while achieving effective target detection.

[0062] Optionally, after obtaining the first target location information in each target video frame, the method further includes:

[0063] Obtain the first target interest range information of the target second video frame, wherein the target second video frame is the video frame preceding the first video frame;

[0064] Target association analysis is performed on the target interest range information of the second video frame and the first target location information of the first video frame to determine the second target location information in the first video frame.

[0065] Specifically, after obtaining the detection results of each keyframe, i.e., the first target location information of the first video frame, the cloud platform compares it with the detection results of the previous second video frame. By referring to the historical records of target movement speed and direction, the targets in the second video frame are identified and associated with the targets in the first video frame. For targets that appear only in the first video frame and not in the previous second video frame, they are marked as "new targets," i.e., the second target location information, and the location range of such targets is fed back to the end device.

[0066] In this embodiment of the application, by performing target association analysis on the target interest range information of the second video frame and the first target location information of the first video frame, it is possible to effectively determine whether the target in the first video frame is a new target or a target from the previous video frame, and new coordinates can be effectively detected.

[0067] Optionally, after determining the second target location information in the first video frame, the method further includes:

[0068] The second target location information is sent to the terminal so that the terminal can determine the second target interest range information in the continuous video frame group where the second video frame of the target is located based on the second target location information and through a target tracking algorithm;

[0069] Obtain the second target interest range information fed back by the terminal.

[0070] Specifically, after obtaining the second target location information fed back by the cloud platform, the terminal device described in this application calls the tracking algorithm with the second target location information as the initial value, and reverse tracks in the second video frame of the previous frame group to obtain the possible location range of the new target, thereby obtaining the second target interest range information in the continuous video frame group.

[0071] The second target interest range information described in this application may contain new targets to be detected. Therefore, in this embodiment, the terminal will re-upload the image information of the second target interest range information to the cloud platform for detection.

[0072] The cloud platform will re-invoke the target detection neural network for such images and merge the obtained target detection results with the results of the first detection at the corresponding time point to obtain a more complete target detection result.

[0073] In this embodiment, a reverse tracking method is used to achieve full detection of newly emerging targets, thereby improving the overall accuracy of the target detection task.

[0074] Optionally, after determining the second target location information in the first video frame, the method further includes:

[0075] Based on the number of second targets in the second target location, determine the occurrence ratio information of the second targets;

[0076] Based on the second target occurrence ratio information, the length information of the continuous video frame group is determined;

[0077] The length information is sent to the terminal so that the terminal can regroup the original video according to the length information to obtain multiple groups of consecutive video frames after regrouping.

[0078] Specifically, after the cloud platform identifies the location information of the second target in each first video frame, it will calculate the number T of targets in the second target location information. N The number T of all detected targets in this frame A Divide the two to obtain the second target occurrence rate information R:

[0079]

[0080] The algorithm will recalculate the frame group length information N′ suitable for the current video recognition difficulty according to the following formula:

[0081]

[0082] Where N is the original length information of the consecutive video frame group, N max and N min These are the maximum and minimum preset lengths of the frame group, R. th1 and R th2 It is a preset occurrence ratio threshold, N′ is the length information of the continuous video frame group, and R is the occurrence ratio information of the second target.

[0083] The cloud platform sends the new frame group length N′ to the end device via the network. After receiving the new N′ value, the end device will re-aggregate all frame groups that have not yet uploaded their first video frame according to the new N′ value, and reassign the first frame of each re-aggregated group as the first video frame and subsequent frames as the second video frames.

[0084] In this embodiment, after each detection of the second target location information, the frame group length is adjusted in real time according to the video content and the difficulty of the target detection task, so as to achieve a dynamic balance between detection accuracy and bandwidth requirements.

[0085] Optionally, in this embodiment, when the computation processing of any frame group is temporarily interrupted due to communication with the cloud device and waiting for the return of the detection result of its first video frame, the end device will select the first frame group that can immediately start image encoding or tracking in chronological order, and use the end computing power to process the selected frame group until the previously waiting frame group ends communication and resumes computation, thereby preventing the end device from being idle and making full use of computing power.

[0086] The target detection device provided by the present invention is described below. The target detection device described below and the target detection method described above can be referred to in correspondence.

[0087] Figure 2 This is a schematic diagram of the target detection device provided by the present invention, as shown below. Figure 2 As shown, it includes: a first acquisition module 210, a detection module 220, and a second acquisition module 230; wherein, the first acquisition module 210 is used to acquire multiple groups of continuous video frames sent by the terminal, wherein each group of continuous video frames includes a first video frame and N second video frames; wherein, the detection module 220 is used to perform target detection on each of the first video frames, obtain the first target location information in each of the first video frames, and send each of the first target location information to the terminal, so that the terminal can determine the first target interest range information of the second video frames in each group of continuous video frames based on each of the first target location information and through a target tracking algorithm; wherein, the second acquisition module 230 is used to acquire the first target interest range information fed back by the terminal.

[0088] Optionally, the device further includes:

[0089] The third acquisition module is used to acquire the first target interest range information of the target second video frame, wherein the target second video frame is the video frame preceding the first video frame.

[0090] The analysis module is used to perform target association analysis on the target interest range information of the second video frame and the first target location information of the first video frame to determine the second target location information in the first video frame.

[0091] Optionally, the device further includes:

[0092] The sending module is used to send the second target location information to the terminal, so that the terminal can determine the second target interest range information in the continuous video frame group where the second video frame of the target is located based on the second target location information and through a target tracking algorithm;

[0093] The fourth acquisition module is used to acquire the second target interest range information fed back by the terminal.

[0094] In this embodiment, after obtaining a continuous group of video frames sent by the terminal, target detection is first performed only on the first video frame to determine the first target location information. Then, based on the first target location information, the terminal retains only the area where the first target location information may exist in the second video frame, that is, only the first target interest range information is retained. During data upload, data in the second video frame other than the first target interest range information is discarded. This can significantly reduce the bandwidth required for video data upload while achieving effective target detection.

[0095] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a target detection method. This method includes: acquiring multiple sets of consecutive video frames sent by a terminal, wherein each set of consecutive video frames includes a first video frame and N second video frames; performing target detection on each of the first video frames to obtain first target location information in each first video frame, and sending each first target location information to the terminal so that the terminal, based on each first target location information, determines the first target interest range information of the second video frames in each set of consecutive video frames using a target tracking algorithm; and acquiring the first target interest range information fed back by the terminal.

[0096] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the target detection method provided by the above methods, the method comprising: acquiring multiple groups of consecutive video frames sent by a terminal, wherein each group of consecutive video frames comprises a first video frame and N second video frames; performing target detection on each of the first video frames to obtain first target location information in each of the first video frames, and sending each of the first target location information to the terminal, so that the terminal, based on each of the first target location information, determines the first target interest range information of the second video frames in each group of consecutive video frames through a target tracking algorithm; and acquiring the first target interest range information fed back by the terminal.

[0098] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the target detection methods provided above. The method includes: acquiring multiple groups of consecutive video frames sent by a terminal, wherein each group of consecutive video frames includes a first video frame and N second video frames; performing target detection on each of the first video frames to obtain first target location information in each of the first video frames, and sending each of the first target location information to the terminal, so that the terminal, based on each of the first target location information, determines first target interest range information of the second video frames in each group of consecutive video frames using a target tracking algorithm; and acquiring the first target interest range information fed back by the terminal.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection method, characterized in that, include: Acquire multiple groups of consecutive video frames sent by the terminal, wherein each group of consecutive video frames includes a first video frame and N second video frames; Target detection is performed on each of the first video frames to obtain the first target location information in each of the first video frames, and the first target location information is sent to the terminal so that the terminal can determine the first target interest range information of the second video frame in each group of consecutive video frames based on the first target location information and through a target tracking algorithm. Obtain the first target interest range information fed back by the terminal; The method further includes, after obtaining the first target location information in each target video frame: Obtain the first target interest range information of the target second video frame, wherein the target second video frame is the video frame preceding the first video frame; Target association analysis is performed on the target interest range information of the second video frame and the first target location information of the first video frame to determine the second target location information in the first video frame; After determining the location information of the second target in the first video frame, the method further includes: Based on the number of second targets in the second target location, determine the occurrence ratio information of the second targets; Based on the second target occurrence ratio information, the length information of the continuous video frame group is determined; The length information is sent to the terminal so that the terminal can regroup the original video according to the length information to obtain multiple groups of consecutive video frames after regrouping. The determination of the length information of the consecutive video frame group based on the second target occurrence ratio information includes: Where N is the original length information of the consecutive video frame group, N max and N min These are the maximum and minimum preset lengths of the frame group, R. th1 and R th2 It is a preset occurrence rate threshold, N ′ R represents the length information of a continuous video frame group, and R represents the occurrence ratio information of the second target.

2. The target detection method according to claim 1, characterized in that, After determining the location information of the second target in the first video frame, the method further includes: The second target location information is sent to the terminal so that the terminal can determine the second target interest range information in the continuous video frame group where the second video frame of the target is located based on the second target location information and through a target tracking algorithm; Obtain the second target interest range information fed back by the terminal.

3. A target detection device, characterized in that, include: The first acquisition module is used to acquire multiple groups of consecutive video frames sent by the terminal, wherein each group of consecutive video frames includes a first video frame and N second video frames. The detection module is used to perform target detection on each of the first video frames, obtain the first target location information in each of the first video frames, and send the first target location information to the terminal so that the terminal can determine the first target interest range information of the second video frame in each group of the consecutive video frames based on the first target location information and through a target tracking algorithm. The second acquisition module is used to acquire the first target interest range information fed back by the terminal; The device further includes: The third acquisition module is used to acquire the first target interest range information of the target second video frame, wherein the target second video frame is the video frame preceding the first video frame. The analysis module is used to perform target association analysis on the target interest range information of the second video frame and the first target location information of the first video frame to determine the second target location information in the first video frame; The device is also used for: Based on the number of second targets in the second target location, determine the occurrence ratio information of the second targets; Based on the second target occurrence ratio information, the length information of the continuous video frame group is determined; The length information is sent to the terminal so that the terminal can regroup the original video according to the length information to obtain multiple groups of consecutive video frames after regrouping. The determination of the length information of the consecutive video frame group based on the second target occurrence ratio information includes: Where N is the original length information of the consecutive video frame group, N max and N min These are the maximum and minimum preset lengths of the frame group, R. th1 and R th2 It is a preset occurrence rate threshold, N ′ R represents the length information of a continuous video frame group, and R represents the occurrence ratio information of the second target.

4. The target detection device according to claim 3, characterized in that, The device further includes: The sending module is used to send the second target location information to the terminal, so that the terminal can determine the second target interest range information in the continuous video frame group where the second video frame of the target is located based on the second target location information and through a target tracking algorithm; The fourth acquisition module is used to acquire the second target interest range information fed back by the terminal.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the target detection method as described in any one of claims 1 to 2.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the target detection method as described in any one of claims 1 to 2.

Images