Video detection methods, devices, electronic equipment and readable storage media

Abstract

This invention provides a video detection method, apparatus, electronic device, and readable storage medium. The method includes: acquiring video data to be detected; acquiring a video segment to be detected that includes a target object from the video data to be detected; determining the pixel gradient features of the target object in the video segment to be detected; and determining whether the video segment to be detected is a target video segment based on the pixel gradient features. The target video segment is used for pose analysis of the target object. This reduces the amount of computational data and computational overhead, improving the efficiency of video detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video detection method, apparatus, electronic device and readable storage medium. Background Technology

[0002] For public safety and security, video analytics systems can be used for intelligent analysis to effectively assist back-office personnel in judging and handling abnormal situations.

[0003] Currently, video analytics requires a large amount of data collection, and the analysis process requires pose analysis of each frame of video image and each target, resulting in high computational overhead and low efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a video detection method, apparatus, electronic device, and readable storage medium, which solves the problem of low efficiency in video analysis in the prior art.

[0005] To address the aforementioned problems, in a first aspect, embodiments of the present invention provide a video detection method, comprising:

[0006] Acquire the video data to be detected;

[0007] Obtain the video segment to be detected that includes the target object from the video data to be detected;

[0008] Determine the pixel gradient features of the target object in the video segment to be detected;

[0009] Based on the pixel gradient features, determine whether the video segment to be detected is the target video segment;

[0010] The target video segment is used to perform pose analysis on the target object.

[0011] Secondly, embodiments of the present invention provide a video detection device, characterized in that it includes:

[0012] The first acquisition module is used to acquire the video data to be detected;

[0013] The second acquisition module is used to acquire video segments containing target objects from the video data to be detected.

[0014] The first determining module is used to determine the pixel gradient features of the target object in the video segment to be detected;

[0015] The second determining module is used to determine whether the video segment to be detected is a target video segment based on the pixel gradient features.

[0016] The target video segment is used to perform pose analysis on the target object.

[0017] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the video detection method described above.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video detection method described above.

[0019] One of the above technical solutions has the following advantages or beneficial effects:

[0020] The video detection method, apparatus, electronic device, and readable storage medium provided in this invention embodiment, wherein the method is based on a video segment containing a target object in the video data to be detected. By determining the pixel gradient features of the target object in the video segment, and based on the pixel gradient features and the specific application scenario, it can be determined whether further pose analysis of the video segment is necessary. In the subsequent pose analysis process, it is not necessary to perform pose analysis on every frame or every object of the original video data; only the video segment identified as the target segment needs to be analyzed. This not only reduces the amount of computational data and computational power overhead, but also avoids wasting time on meaningless video segments, greatly improving the efficiency of video detection. This allows backend personnel to promptly perceive the behavior of the target object in the video data, which is particularly beneficial for timely prevention and control of abnormal behavior of the target object. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the video detection method provided in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram illustrating the principle of determining a block to be detected according to an embodiment of the present invention;

[0023] Figure 3a This is a schematic diagram of a block to be detected according to an embodiment of the present invention;

[0024] Figure 3b This is a schematic diagram illustrating the principle of determining the gradient of the first pixel according to an embodiment of the present invention;

[0025] Figure 4 This is a structural diagram of a network system applicable to an embodiment of the present invention;

[0026] Figure 5This is a schematic diagram illustrating the principle of a video detection method provided in an embodiment of the present invention;

[0027] Figure 6 This is a structural diagram of the video detection device provided in an embodiment of the present invention;

[0028] Figure 7 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.

[0031] In embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0032] Please see Figure 1 , Figure 1 This is a flowchart illustrating a video detection method provided in an embodiment of the present invention.

[0033] like Figure 1 As shown, the video detection method includes the following steps:

[0034] Step 101: Obtain the video data to be detected.

[0035] The video data to be detected refers to the raw video data acquired by the image acquisition device. It can be real-time video stream data or pre-acquired video data. The image acquisition device can be a camera, camcorder, or other device with shooting capabilities, such as a mobile phone or tablet. Cameras can include cameras from various monitoring systems, and the specific type can be determined according to the actual situation, without limitation here.

[0036] In practice, the video data to be detected can be acquired through acquisition. For example, if the image acquisition device is a terminal with sufficient computing power, it can execute the video detection method and acquire the video data to be detected. Alternatively, the video data can be acquired through reception. For example, an edge device or cloud device can execute the video detection method and receive the video data to be detected sent by the image acquisition device. It can also be acquired by calling an interface. For example, if the image acquisition device lacks communication capabilities, an edge device or cloud device can execute the video detection algorithm and call an interface to acquire the video data to be detected. The specific implementation depends on the actual situation and is not limited here.

[0037] Step 102: Obtain the video segment to be detected that includes the target object from the video data to be detected.

[0038] In this step, since the video data to be detected is raw video data, it may contain a large number of video segments without any objects. Based on this, we can first filter out the video segments in the video data to be detected that include the target object as the subsequent video segments to be detected, thereby reducing the amount of detection data.

[0039] In practical implementation, target detection algorithms can be used to identify target objects in the video data to be detected. Target objects can be people or animals, depending on the actual scenario. For example, in a surveillance scenario, the target object can be a person to detect whether there are situations such as physical conflict or people falling in the surveillance video. The target detection algorithm can be a commonly used target detection algorithm in related technologies, such as inter-frame differencing, background subtraction, and optical flow. For details, please refer to the descriptions of the algorithms in related technologies; they will not be elaborated upon here.

[0040] After identifying the target object, a target tracking algorithm can be used to track the target object, and based on the target object's motion trajectory, all video frames in which the target object appears can be extracted as the video segment to be detected. The target tracking algorithm can be a commonly used target tracking algorithm in related technologies, such as the IOU tracking algorithm (Intersection-over-union Tracker, IOU Tracker), etc. For details, please refer to the description of the algorithm in related technologies, which will not be elaborated here.

[0041] When there are multiple target objects, the video segment to be detected for each target object can be obtained separately. For example, if pedestrian A and pedestrian B appear in the video data to be detected, then the video segment to be detected corresponding to pedestrian A can be obtained as segment 1 and the video segment to be detected corresponding to pedestrian B can be obtained separately. If the target object appears multiple times at intervals in the video data to be detected, the video segment to be detected for each appearance can be obtained separately. For example, if pedestrian A appears in the video data to be detected at 03:00-03:30 and 04:50-05:10, then the video segment to be detected corresponding to pedestrian A at 03:00-03:30 can be obtained as segment 1.1 and the video segment to be detected corresponding to pedestrian A at 04:50-05:10 can be obtained separately. The specific method depends on the actual situation and is not limited here.

[0042] Step 103: Determine the pixel gradient features of the target object in the video segment to be detected.

[0043] Pixel gradient features are used to characterize the changing trends of each pixel within a target object. They can be determined based on the feature information of each pixel within the target object, which may include, but is not limited to, grayscale features, brightness features, and color features. Since the target object appears in every frame of the video segment to be detected, and the video frames are arranged in chronological order, the target objects in different video frames can be superimposed to form a three-dimensional image space based on the time series of the video frames. Then, the pixel gradient features can be determined based on the feature information of each pixel within the target object.

[0044] Step 104: Determine whether the video segment to be detected is the target video segment based on the pixel gradient features.

[0045] Among them, the target video clip is used to perform pose analysis on the target object.

[0046] Based on pixel gradient features, the changing trends of each pixel in a target object can be determined, that is, the rate of change of each pixel along each direction. Based on this, the positional changes of the entire or partial body of the target object can be determined based on the changes in pixels, and thus the movement trend of the target object can be predicted.

[0047] In specific judgments, a judgment condition can be preset. This condition can be determined based on the application scenario of the subsequent posture analysis, considering factors such as the motion trajectory, amplitude, and kinetic energy of the target object or a portion of its body. The specific condition is determined according to the actual situation and is not limited here. For example, if the posture analysis of the target object is used to determine whether the target object exhibits large-scale limb movements such as fighting or falling, then corresponding conditions can be set for the pixel gradient features to indicate that the target object exhibits large-scale limb movements. Therefore, if the pixel gradient features characterize the video segment to be detected as meeting the judgment condition, the video segment to be detected can be identified as the target video segment, and posture analysis of the target object can then be performed based on the target video segment.

[0048] It should be noted that the entity executing this video detection method can continue to perform pose analysis on the target object based on the target video segment, or it can output the target video segment so that other devices can perform pose analysis on the target object based on the target video segment. Specific implementation methods for pose analysis can be found in related technical documents and are not specifically limited here.

[0049] In this embodiment of the invention, based on a video segment containing a target object within the video data to be detected, the pixel gradient features of the target object in the video segment are determined. Based on these pixel gradient features and the specific application scenario, it can be determined whether further pose analysis of the video segment is necessary. In subsequent pose analysis, it is unnecessary to perform pose analysis on every frame or every object in the original video data; only the video segment identified as the target segment needs to be analyzed. This not only reduces the amount of computational data and computational overhead but also avoids wasting time on meaningless video segments, greatly improving the efficiency of video detection. This allows backend personnel to promptly perceive the behavior of the target object in the video data, which is particularly beneficial for timely prevention and control of abnormal behavior of the target object.

[0050] Optionally, determining the pixel gradient features of the target object in the video segment to be detected includes:

[0051] Identify N blocks to be detected corresponding to the target object, where N is a positive integer;

[0052] Based on the time series of each video frame in the video segment to be detected, determine the first pixel gradient of each of the N blocks to be detected in each of the M preset directions, where M is a positive integer;

[0053] Based on the time series of each video frame, determine the second pixel gradient of the target object in each of the M preset directions;

[0054] The pixel gradient features include the first pixel gradient and the second pixel gradient.

[0055] In this embodiment, to further determine the motion trend of the target object's local body, N detection blocks and M preset directions can be defined for the target object. For the target object in each video frame of the video segment to be detected, N detection blocks are defined accordingly. Detection blocks at corresponding positions in different video frames can be superimposed based on the time series of the video frames to determine the first pixel gradient of each detection block in each preset direction. In addition, target regions in different video frames can be superimposed based on the time series of the video frames to determine the second pixel gradient of the target object in each preset direction. Subsequently, based on the first pixel gradient and the second pixel gradient, it can be determined whether the video segment to be detected is the target video segment.

[0056] The N blocks to be detected may include one or more blocks on the target object. Each block corresponds to a body part of the target object. The size of each block can be the same or different, depending on the actual situation of the target object. For example, Figure 2 As shown, target object 1 is a person. There are six areas 11 to be detected on target object 1, which correspond to the person's head, left hand, right hand, core torso, left foot, and right foot respectively.

[0057] In one optional implementation, N blocks to be detected corresponding to the target object are determined, including:

[0058] Determine the target region corresponding to the target object, and determine k first blocks to be detected within the target region, where k is a positive integer;

[0059] Determine at least one adjacent region of the target region, and in each adjacent region determine k second blocks to be detected corresponding to k first blocks to be detected. The N blocks to be detected include the k first blocks to be detected and the k second blocks to be detected in each adjacent region.

[0060] In this embodiment, considering that the position of the target object, especially the local body of the target object, may change significantly between adjacent frames, in order to completely obtain the motion trajectory of the target object and / or the local body of the target object and accurately determine the motion trend of the target object and / or the local body of the target object, the N blocks to be detected include not only the k first blocks to be detected in the target area corresponding to the target object, but also the k second blocks to be detected in each adjacent area of ​​the target area. It can be understood that the number and size of the second blocks to be detected in each adjacent block are the same as those of the first blocks to be detected.

[0061] In practice, a target region can be defined first, encompassing the target object. This target region can be an irregular area bounded by the target object's boundary, or it can be a minimum rectangular area containing the target object, depending on the specific circumstances. Next, based on the target object's body features, k first detection blocks are determined within the target region, each corresponding to a local part of the target object's body. Then, one or more adjacent regions can be extended outwards from the edge of the target region. Each adjacent region can have the same or different dimensions as the target region, and these one or more adjacent regions enclose the target region. In a preferred embodiment, the target region is rectangular, with eight adjacent regions, each having the same size as the target region, and all eight adjacent regions enclosing the target region. Finally, within each adjacent region, k second detection blocks with the same size and corresponding positions as the k first detection blocks can be determined.

[0062] To make it easier to understand, here is an example: Figure 2 As shown, a target region 2 is defined based on the target object 1, and six blocks 11 to be detected are determined within the target region 2, corresponding to the head, left hand, right hand, core torso, left foot, and right foot, respectively. Then, eight adjacent regions 3 can be determined for the target region 2. Within each adjacent region 3, six second blocks 31 of the same size and position as the six blocks 11 to be detected can be determined. Thus, the number of blocks to be detected for the target object 1 is 6 * 9 = 54.

[0063] The M preset directions can be defined by the user and can be determined according to the actual situation; no specific limitation is made here. In one optional embodiment, the M preset directions include eight directions: directly above, upper right, directly right, lower right, directly below, lower left, directly left, and upper left. That is, for each block to be detected, at least eight first pixel gradients can be determined. Similarly, the second pixel gradient of the target object in each direction can be determined separately; that is, for the target object, at least eight second pixel gradients can be determined.

[0064] In one optional implementation, based on the time series of each video frame in the video segment to be detected, the first pixel gradient of each of the N blocks to be detected in each of the M preset directions is determined, including:

[0065] Obtain the HSV value of the target detection block in each video frame. The target detection block is any block among N detection blocks.

[0066] Based on the HSV values ​​of the target block to be detected in the i-th video frame and the (i+1)-th video frame, the first pixel gradient of the target block to be detected in the target direction is determined. The target direction is any one of the M preset directions.

[0067] In this embodiment, the first pixel gradient of the target detection block in the target direction is used as an example for explanation. The first pixel gradient of other detection blocks and / or other directions can be described with reference to this embodiment. Specifically, a color histogram in HSV space can be established for each video frame image to obtain the HSV value of the target detection block in each video frame. HSV values ​​can better reflect the intuitive characteristics of the color features of pixels. Determining the first pixel gradient based on the HSV value can more accurately reflect the changing trend of pixels within the target detection block.

[0068] The HSV value of the target detection block includes the HSV values ​​of each pixel in the target detection block. The HSV value of a pixel includes its H (hue), S (saturation), and V (luminance) values. In specific implementations, the first pixel gradient of the target detection block in the target direction can be determined based on the H, S, and V values ​​respectively, and then the average of the three first pixel gradients is taken to obtain the final first pixel gradient of the target detection block. Alternatively, the H, S, and V values ​​of each pixel can be normalized to obtain the feature values ​​of each pixel, and then the first pixel gradient of the target detection block in the target direction can be determined based on the feature values. In an optional implementation, average pooling can be performed on the target detection block to obtain the HSV values ​​of each pixel in the target detection block. The specific implementation can be determined according to the actual situation and is not specifically limited here.

[0069] In this embodiment, the target detection blocks in different video frames can be superimposed based on the time series of each video frame in the video segment to be detected. Then, a first pixel gradient is determined based on the HSV values ​​of the target detection blocks in any two adjacent video frames. It should be noted that the specific implementation of determining the first pixel gradient based on the HSV values ​​of the target detection blocks can be referred to the description of related technologies, and is not specifically limited here.

[0070] To make it easier to understand, here is an example: Figure 3a As shown, the pixel size of the target detection block is 5*3. Based on the time series of each video frame in the video segment to be detected, the target detection blocks in different video frames are superimposed, as shown... Figure 3bAs shown, assuming the two adjacent video frames are the first video frame 4 and the second video frame 5, the gradient can be calculated from the center pixel of the first video frame 4 along eight directions: directly above, to the upper right, to the right, to the lower right, to the lower right, to the lower left, to the lower left, and to the upper left of the second video frame 5. This yields eight first pixel gradients. If the video segment to be detected comprises q video frames, then the target block to be detected can have 8*(q-1) first pixel gradients.

[0071] In one alternative implementation, determining the second pixel gradient of the target object in the target direction based on the time series of each video frame includes:

[0072] Obtain the HSV value of the target object in each video frame;

[0073] The second pixel gradient of the target object in the target direction is determined based on the HSV value of the target object in the j-th video frame and the HSV value of the target object in the (j+1)-th video frame.

[0074] In this embodiment, the description of the first pixel gradient of the target block to be detected in the target direction can be referred to above. To avoid repetition, it will not be repeated here.

[0075] In one optional implementation, determining whether a video segment to be detected is a target video segment based on pixel gradient features includes:

[0076] Based on the quotient of the first pixel gradient divided by the second pixel gradient, determine the relative pixel gradient of each of the N blocks to be detected in each of the M preset directions.

[0077] Based on the relative pixel gradient, determine whether the video segment to be detected is the target video segment.

[0078] In this embodiment, the relative pixel gradient of each detection block in each preset direction can be used to determine the motion trend of each detection block relative to the target object as a whole. When the target object is a person, the motion trend of a local body relative to the person as a whole can be determined. For example, the motion trend of the left hand relative to the person as a whole can determine whether the person is making a punch or waving motion. Similarly, the motion trend of the left or right foot relative to the person as a whole can determine whether the person is making a kicking or running motion. Furthermore, the motion trend of the core torso relative to the person as a whole can determine whether the person is making a forward or backward leaning motion.

[0079] In practice, the relative pixel gradient can be equal to the quotient of the first pixel gradient divided by the second pixel gradient, or it can be determined by considering other influencing factors based on the quotient of the first pixel gradient divided by the second pixel gradient. The specific determination can be made according to the actual situation, and no specific limitation is made here.

[0080] Optionally, determining whether a video segment to be detected is a target video segment based on relative pixel gradients can include two implementation methods:

[0081] In the first implementation, the video segment to be detected is determined as the target video segment based on the relative pixel gradient, provided that the relative pixel gradient meets a first preset condition. The first preset condition may be that the relative pixel gradient exceeds a relative gradient threshold, or that the relative pixel gradient is within the range of the relative gradient threshold. The specific condition can be determined according to the actual situation and is not specifically limited here.

[0082] In the second implementation, based on the relative pixel gradient, other influencing factors are considered to comprehensively determine whether the second preset condition is met. Optionally, the other influencing factors may be at least one of the pixel size and weight coefficient of each block to be detected. Specifically, before determining whether the video segment to be detected is the target video segment based on the relative pixel gradient, the video detection method further includes:

[0083] Determine at least one of the pixel size and weight coefficient for each of the N blocks to be detected;

[0084] The step of determining whether a video segment to be detected is a target video segment based on relative pixel gradient includes:

[0085] Based on at least one of the pixel size and weight coefficient of each of the N blocks to be detected, and the relative pixel gradient, determine the block kinetic index of each of the N blocks to be detected in each of the M directions;

[0086] If the block kinetic index exceeds a preset threshold, the video segment to be detected is identified as the target video segment.

[0087] In this embodiment, considering the pixel size and / or weight coefficient of each detection block, as well as the relative pixel gradient, the block kinetic energy index of each detection block relative to the target object can be determined. The block kinetic energy index characterizes the energy possessed by each pixel within the block due to motion, which can be understood analogously to the kinetic energy of an object: for the kinetic energy of an object, the greater the mass, the greater the kinetic energy; the greater the speed, the greater the kinetic energy. For the block kinetic energy index, it can be understood that the greater the relative pixel gradient of the block, the greater the rate of change of the pixels within the block, and thus the greater the block kinetic energy index; the greater the weight coefficient of the block, the greater the weight of the block, and thus the greater the block kinetic energy index; the larger the pixel size of the block, the smaller the motion trend of the block, and the smaller the kinetic energy of the block. It can be understood that, under the same relative pixel gradient, the kinetic energy of the block corresponding to the fist is greater than the kinetic energy of the block corresponding to the core torso.

[0088] It should be noted that determining the video segment to be detected as the target video segment when the block kinetic index exceeds the preset threshold means that when at least one of the N blocks to be detected has a block kinetic index exceeding the preset threshold in at least one direction, the video segment to be detected can be determined as the target video segment.

[0089] Furthermore, optionally, the formula for calculating the block kinetic index is:

[0090]

[0091] in, Let N(s-α, σ) be the block kinetic energy index of the block n to be detected in direction d. 2 Let s and σ be the normal distribution function. 2 Let be the normal distribution constant, and α be the adjustment coefficient used to adjust the weight of the block n to be detected. Let S be the relative pixel gradient of the block n to be detected in the direction d, and let S be the pixel size of the block n to be detected.

[0092] In this embodiment, based on the above-described determination of the block kinetic energy index of each block to be detected, N(s-α, σ) can be adjusted by adjusting α. 2 The peak value of ) is used to adjust the weight coefficient of the block to be detected n accordingly. Here, n is the identifier of the block to be detected, and d is the identifier of the preset direction. For example, the block kinetic energy index of block 1 in direction 1... for:

[0093]

[0094] The following describes the network systems to which the embodiments of the present invention can be applied:

[0095] In this embodiment of the invention, the applicable network systems may optionally include the following five types:

[0096] The first type involves a network system that only includes cloud-side devices.

[0097] Cloud-side devices can be cloud-side computers, servers, or other devices or data platforms with data processing capabilities, depending on the actual situation. This embodiment of the invention does not impose any limitations on this. In this network system, the image acquisition device is not a smart terminal and cannot communicate with the cloud-side device. The cloud-side device can obtain the video data to be detected through interface calls and execute the video detection method provided in this embodiment of the invention to determine whether the video segment to be detected is a target video segment. If it is a target video segment, the device performs pose estimation on the video segment to be detected.

[0098] The second type is a network system that only includes terminals.

[0099] The terminal can also be called user equipment (UE), which can be a camera, camcorder, camera, or other device with shooting function such as mobile phone, tablet, in-vehicle device, wearable device, etc. In-vehicle device includes dashcam, and wearable device includes wristband, glasses, etc. The specific device can be determined according to the actual situation, and the embodiments of the present invention do not limit it.

[0100] In this network system, the image acquisition device is an intelligent terminal with a certain computing power. The terminal can acquire the video data to be detected and execute the video detection method provided in this embodiment of the invention to determine whether the video segment to be detected is a target video segment. If it is a target video segment, the terminal performs pose estimation on the video segment to be detected.

[0101] The third type is a network system that includes edge-side devices, or includes both edge-side devices and cloud-side devices.

[0102] Edge-side devices can be computers, servers, or other data processing devices or data platforms capable of edge computing. The specific choice depends on the actual situation, and this embodiment of the invention does not impose any limitations on this. In this network system, to improve computing efficiency and reduce network overhead, edge-side computing resources are introduced. Specifically, based on the edge-side computing power and the frame rate requirements of the detection scenario, it can be determined whether to deploy only an edge-side network or a collaborative edge-cloud network (hereinafter referred to as a cloud-edge collaborative network).

[0103] In this network system, the image acquisition device is not a smart terminal and cannot communicate with edge devices or cloud devices.

[0104] In the case of deploying only the edge-side network, the edge-side device can obtain the video data to be detected through interface calls, and execute the video detection method provided in the embodiments of the present invention to determine whether the video segment to be detected is a target video segment, and if it is a target video segment, perform pose estimation on the video segment to be detected.

[0105] In an optional implementation of a cloud-edge collaborative network, after determining whether a video segment to be detected is a target video segment based on pixel gradient features, the video detection method further includes: if the video segment to be detected is determined to be a target video segment, sending the video segment to be detected to the cloud-side device. That is, the edge-side device can obtain the video data to be detected through an interface call, execute the video detection method provided in this embodiment of the invention, determine whether the video segment to be detected is a target video segment, and if it is a target video segment, send the video segment to be detected to the cloud-side device so that the cloud-side device can perform pose estimation. Alternatively, in another optional implementation, the edge-side device can obtain the video data to be detected through an interface call, execute the video detection method provided in this embodiment of the invention, determine whether the video segment to be detected is a target video segment, and if it is a target video segment, extract features for pose analysis from the video segment to be detected, and send the extracted feature map data to the cloud-side device so that the cloud-side device can perform further pose estimation. The specific implementation can be determined according to the actual situation, and this embodiment of the invention does not impose specific limitations here.

[0106] The fourth type is a network system that includes terminals and edge devices.

[0107] In this network system, edge computing resources are introduced to improve computational efficiency and reduce network overhead. The image acquisition device is an intelligent terminal with a certain computing power and can communicate with edge devices.

[0108] Optionally, after determining whether the video segment to be detected is a target video segment based on pixel gradient features, the video detection method further includes: if the video segment to be detected is determined to be a target video segment, sending the video segment to be detected to the edge-side device. That is, the terminal can collect and acquire the video data to be detected, execute the video detection method provided in this embodiment of the invention, determine whether the video segment to be detected is a target video segment, and if it is a target video segment, send the video segment to be detected to the edge-side device so that the edge-side device can perform pose estimation. This also ensures that the video data does not leave the domain, improving user privacy and security.

[0109] The fifth type is a network system that includes terminals, edge devices, and cloud devices.

[0110] In this network system, to further improve computational efficiency and reduce network overhead, edge computing resources and cloud computing resources are introduced. For example... Figure 4 As shown, the image acquisition device is an intelligent terminal with a certain computing power and can communicate with edge devices, thus realizing a cloud-edge-device collaborative distributed network.

[0111] In practical implementation, multiple network segmentation layers can be determined based on the existing hardware environment of edge devices, such as the computing power and throughput of the central processing unit (CPU). The segmented lower-level networks are then deployed on the edge to ensure that the lower-level networks can fully utilize the computing power of the edge and reduce computing power consumption. The terminal can collect and acquire the video data to be detected and execute the video detection method provided in this embodiment of the invention to determine whether the video segment to be detected is a target video segment. If it is a target video segment, the terminal sends the video segment to be detected to the edge devices. Each edge device can correspond to one or more terminals and acquire the target video segment sent by the terminal for feature extraction. The cloud-side device can acquire feature map data from one or more edge devices for further pose estimation and inference.

[0112] An exemplary implementation method of the present invention is described below:

[0113] The network system used in this embodiment is the fifth type of network system described above, such as... Figure 5 As shown:

[0114] Step 1: The terminal collects the video data to be tested.

[0115] In this step, the terminal collects the video data to be tested through the camera.

[0116] Step 2: The terminal screens target video segments using the block kinetic energy index.

[0117] In this step, the terminal extracts multi-scale detection blocks from the target object to distinguish the fluctuation trends of pixels within the target object and its corresponding detection blocks, thereby determining the block kinetic index of the target object and its corresponding detection blocks. If the block kinetic index meets preset conditions, the video segment to be detected is determined as the target video segment. The specific process is as follows:

[0118] 1) The terminal uses an object detection algorithm to identify the target pedestrian in the video data to be detected and delineates the target area. The target area is the smallest rectangular area containing the target object. Then, the IOU tracking algorithm is used to track the target pedestrian, extract the video segment to be detected that contains the target pedestrian in the video data to be detected, and determine the target area on each video frame in the video segment to be detected.

[0119] 2) The terminal uses the length direction of the target area as a reference and, based on the body features of the target pedestrian, determines multiple first detection blocks of different scales within the target area. These multiple first detection blocks can correspond to multiple parts of the target pedestrian's body, such as the left palm, right palm, head, core torso, left foot, and right foot. Adjacent areas are added around the target area, and second detection blocks of the same scale corresponding to the positions of the first detection blocks are determined within each adjacent area. Therefore, the target object's detection blocks include multiple first detection blocks within the target area and multiple second detection blocks within each adjacent area.

[0120] 3) The terminal converts the video frame images of the video segment to be detected into HSV images, and uses average pooling to average the values ​​of each block to be detected, obtaining the HSV value of each block. Then, based on the time series of each video frame in the video segment, each block is superimposed to calculate the first pixel gradient of each block in eight preset directions. Furthermore, based on the time series of each video frame in the video segment, the target region is superimposed to calculate the second pixel gradient of the target region in eight preset directions. The eight preset directions are: top, top right, right, bottom right, bottom, bottom left, left, and top left. Then, for any block, the quotient obtained by dividing the first pixel gradient of the block in the target direction by the second pixel gradient of the target region in the target direction is determined as the relative pixel gradient of the block in the target direction. Where n is the block number to be detected, and d is the direction number.

[0121] 4) The terminal weights each block to be detected using a normal distribution.

[0122] The block kinetic energy index of the block n to be detected in direction d is:

[0123]

[0124] in, Let N(s-α, σ) be the block kinetic energy index of the block n to be detected in direction d. 2 Let s and σ be the normal distribution function. 2 Let be the normal distribution constant, and α be the adjustment coefficient used to adjust the weight of the block n to be detected. Let α be the relative pixel gradient of the block n to be detected in direction d, and S be the pixel size of the block n to be detected. By adjusting α, the weight coefficient of the block n to be detected can be adjusted, thereby determining the block kinetic index of the block n to be detected in direction d.

[0125] 5) If the block kinetic index of at least one block to be detected exceeds a preset threshold in at least one direction, the terminal can determine that the video segment to be detected is the target video segment.

[0126] Step 3: The terminal sends the video segment to be detected, which is identified as the target video segment, to the edge device for feature extraction.

[0127] Step 4: The edge device sends the obtained feature map data to the cloud device for further pose estimation and inference.

[0128] See Figure 6 , Figure 6 This is a structural diagram of the video detection device provided in an embodiment of the present invention.

[0129] like Figure 6 As shown, the video detection device 600 includes:

[0130] The first acquisition module 601 is used to acquire the video data to be detected;

[0131] The second acquisition module 602 is used to acquire video segments that include target objects from the video data to be detected.

[0132] The first determining module 603 is used to determine the pixel gradient features of the target object in the video segment to be detected;

[0133] The second determining module 604 is used to determine whether the video segment to be detected is the target video segment based on pixel gradient features.

[0134] Among them, the target video clip is used to perform pose analysis on the target object.

[0135] Optionally, the first determining module 603 includes:

[0136] The first determining unit is used to determine the N blocks to be detected corresponding to the target object, where N is a positive integer;

[0137] The second determining unit is used to determine the first pixel gradient of each of the N blocks to be detected in each of the M preset directions based on the time series of each video frame in the video segment to be detected, where M is a positive integer.

[0138] The third determining unit is used to determine the second pixel gradient of the target object in each of the M preset directions based on the time series of each video frame.

[0139] The pixel gradient features include the first pixel gradient and the second pixel gradient.

[0140] Optionally, the first determining unit includes:

[0141] The first determining subunit is used to determine the target region corresponding to the target object, and to determine k first detection blocks within the target region, where k is a positive integer;

[0142] The second determining subunit is used to determine at least one adjacent region of the target region, and in each adjacent region to determine k second detectable blocks corresponding to k first detectable blocks;

[0143] Among them, the N blocks to be detected include k first blocks to be detected, and k second blocks to be detected in each adjacent region.

[0144] Optionally, the second determining unit includes:

[0145] The first acquisition subunit is used to acquire the HSV value of the target detection block in each video frame. The target detection block is any block among N detection blocks.

[0146] The third determining subunit is used to determine the first pixel gradient of the target detection block in the target direction based on the HSV value of the target detection block in the i-th video frame and the HSV value of the target detection block in the (i+1)-th video frame. The target direction is any one of the M preset directions.

[0147] Optionally, the third determining unit includes:

[0148] The second acquisition subunit is used to acquire the HSV value of the target object in each video frame;

[0149] The fourth determining subunit is used to determine the second pixel gradient of the target object in the target direction based on the HSV value of the target object in the j-th video frame and the HSV value of the target object in the (j+1)-th video frame. The target direction is any one of the M preset directions.

[0150] Optionally, the second determining module 604 includes:

[0151] The fourth determining unit is used to determine the relative pixel gradient of each of the N blocks to be detected in each of the M preset directions based on the quotient of the first pixel gradient divided by the second pixel gradient.

[0152] The fifth determining unit is used to determine whether the video segment to be detected is the target video segment based on the relative pixel gradient.

[0153] Optionally, the video inspection device 600 also includes:

[0154] The third determining module is used to determine at least one of the pixel size and weight coefficient of each of the N blocks to be detected;

[0155] The fifth determining unit includes:

[0156] The fifth determining subunit is used to determine the block kinetic energy index of each of the N blocks to be detected in each of the M directions based on at least one of the pixel size and weight coefficient of each block in the N blocks to be detected, and the relative pixel gradient.

[0157] The sixth determining subunit is used to determine the video segment to be detected as the target video segment when the block kinetic energy index exceeds a preset threshold.

[0158] Optionally, the formula for calculating the block kinetic index is:

[0159]

[0160] in, Let N(s-α, σ) be the block kinetic energy index of the block n to be detected in direction d. 2 Let s and σ be the normal distribution function. 2 Let be the normal distribution constant, and α be the adjustment coefficient used to adjust the weight of the block n to be detected. Let S be the relative pixel gradient of the block n to be detected in the direction d, and let S be the pixel size of the block n to be detected.

[0161] Optionally, the video inspection device 600 also includes:

[0162] The sending module is used to send the video segment to be detected to the edge device or the cloud device when it is determined that the video segment to be detected is the target video segment.

[0163] The video detection device 600 can implement the method embodiments of the present invention. Figure 1 The various processes that can be implemented in the corresponding method embodiments, and the same beneficial effects, will not be described again here to avoid repetition.

[0164] This invention also provides an electronic device. Please refer to [link to relevant documentation]. Figure 7 The electronic device 700 may include a processor 701, a memory 702, and a computer program 7021 stored in the memory 702 and executable on the processor 701. When the computer program 7021 is executed by the processor 701, it can achieve... Figure 1 Any steps in the corresponding method embodiments and the achievement of the same beneficial effects will not be repeated here.

[0165] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by hardware related to program instructions, and the program can be stored in a computer-readable medium. The present invention also provides a computer-readable storage medium storing a third computer program, which, when executed by a fourth processor, can implement the above-described methods. Figure 1 Any step in the corresponding method embodiment can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0166] The storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0167] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A video detection method, characterized in that, include: Acquire the video data to be detected; Obtain the video segment to be detected that includes the target object from the video data to be detected; Determine the pixel gradient features of the target object in the video segment to be detected; Based on the pixel gradient features, determine whether the video segment to be detected is the target video segment; The target video segment is used to perform pose analysis on the target object; Determining the pixel gradient features of the target object in the video segment to be detected includes: Determine N blocks to be detected corresponding to the target object, where N is a positive integer; Based on the time series of each video frame in the video segment to be detected, the first pixel gradient of each of the N blocks to be detected is determined in each of the M preset directions, where M is a positive integer. Based on the time series of each video frame, determine the second pixel gradient of the target object in each of the M preset directions; The pixel gradient features include the first pixel gradient and the second pixel gradient; The step of determining whether the video segment to be detected is the target video segment based on the pixel gradient features includes: Based on the quotient of the first pixel gradient divided by the second pixel gradient, the relative pixel gradient of each of the N blocks to be detected is determined in each of the M preset directions; Based on the relative pixel gradient, it is determined whether the video segment to be detected is the target video segment.

2. The method according to claim 1, characterized in that, The step of determining the N blocks to be detected corresponding to the target object includes: Determine the target region corresponding to the target object, and determine k first detection blocks within the target region, where k is a positive integer; Determine at least one adjacent region of the target region, and in each of the adjacent regions, determine k second blocks to be detected corresponding to the k first blocks to be detected; The N blocks to be detected include the k first blocks to be detected, and the k second blocks to be detected in each of the adjacent regions.

3. The method according to claim 1, characterized in that, The step of determining the first pixel gradient of each of the N blocks to be detected in each of the M preset directions based on the time series of each video frame in the video segment to be detected includes: Obtain the HSV value of the target detection block in each video frame, wherein the target detection block is any block among the N detection blocks; Based on the HSV value of the target detection block in the i-th video frame and the HSV value of the target detection block in the (i+1)-th video frame, the first pixel gradient of the target detection block in the target direction is determined, where the target direction is any one of the M preset directions.

4. The method according to claim 1, characterized in that, The step of determining the second pixel gradient of the target object in each of the M preset directions based on the time series of each video frame includes: Obtain the HSV value of the target object in each video frame; Based on the HSV value of the target object in the j-th video frame and the HSV value of the target object in the (j+1)-th video frame, the second pixel gradient of the target object in the target direction is determined, where the target direction is any one of the M preset directions.

5. The method according to claim 1, characterized in that, Before determining whether the video segment to be detected is the target video segment based on the relative pixel gradient, the method further includes: Determine at least one of the pixel size and weight coefficient for each of the N blocks to be detected; The step of determining whether the video segment to be detected is the target video segment based on the relative pixel gradient includes: Based on at least one of the pixel size and weight coefficient of each of the N blocks to be detected, and the relative pixel gradient, determine the block kinetic index of each of the N blocks to be detected in each of the M directions; If the block kinetic energy index exceeds a preset threshold, the video segment to be detected is determined to be the target video segment.

6. The method according to claim 5, characterized in that, The formula for calculating the block kinetic energy index is: ； in, Let be the block kinetic energy index of the block n to be detected in direction d. Let s and be normal distribution functions. It is the normal distribution constant. This is an adjustment coefficient used to adjust the weight of the block n to be detected. Let S be the relative pixel gradient of the block n to be detected in the direction d, and let S be the pixel size of the block n to be detected.

7. The method according to claim 1, characterized in that, After determining whether the video segment to be detected is the target video segment based on the pixel gradient features, the method further includes: If the video segment to be detected is determined to be the target video segment, the video segment to be detected is sent to the edge device or the cloud device.

8. A video detection device, characterized in that, include: The first acquisition module is used to acquire the video data to be detected; The second acquisition module is used to acquire video segments containing target objects from the video data to be detected. The first determining module is used to determine the pixel gradient features of the target object in the video segment to be detected; The second determining module is used to determine whether the video segment to be detected is a target video segment based on the pixel gradient features. The target video segment is used to perform pose analysis on the target object; The first determining module includes: The first determining unit is used to determine N blocks to be detected corresponding to the target object, where N is a positive integer; The second determining unit is used to determine the first pixel gradient of each of the N blocks to be detected in each of the M preset directions based on the time sequence of each video frame in the video segment to be detected, where M is a positive integer; The third determining unit is used to determine the second pixel gradient of the target object in each of the M preset directions based on the time series of each video frame; The pixel gradient features include the first pixel gradient and the second pixel gradient; The second determining module includes: The fourth determining unit is used to determine the relative pixel gradient of each of the N blocks to be detected in each of the M preset directions based on the quotient of the first pixel gradient divided by the second pixel gradient; The fifth determining unit is used to determine whether the video segment to be detected is a target video segment based on the relative pixel gradient.

9. The apparatus according to claim 8, characterized in that, The first determining unit includes: The first determining subunit is used to determine the target region corresponding to the target object, and to determine k first detection blocks within the target region, where k is a positive integer; The second determining subunit is used to determine at least one adjacent region of the target region, and in each adjacent region determine k second detectable blocks corresponding to the k first detectable blocks; The N blocks to be detected include the k first blocks to be detected, and the k second blocks to be detected in each of the adjacent regions.

10. The apparatus according to claim 8, characterized in that, The second determining unit includes: The first acquisition subunit is used to acquire the HSV value of the target detection block in each video frame, wherein the target detection block is any block among the N detection blocks; The third determining subunit is used to determine the first pixel gradient of the target detection block in the target direction based on the HSV value of the target detection block in the i-th video frame and the HSV value of the target detection block in the (i+1)-th video frame, wherein the target direction is any one of the M preset directions.

11. The apparatus according to claim 8, characterized in that, The third determining unit includes: The second acquisition subunit is used to acquire the HSV value of the target object in each video frame; The fourth determining subunit is used to determine the second pixel gradient of the target object in the target direction based on the HSV value of the target object in the j-th video frame and the HSV value of the target object in the (j+1)-th video frame, wherein the target direction is any one of the M preset directions.

12. The apparatus according to claim 8, characterized in that, The device further includes: The third determining module is used to determine at least one of the pixel size and weight coefficient of each of the N blocks to be detected; The fifth determining unit includes: The fifth determining subunit is used to determine the block kinetic energy index of each of the N blocks to be detected in each of the M directions based on at least one of the pixel size and weight coefficient of each block in the N blocks to be detected, and the relative pixel gradient. The sixth determining subunit is used to determine the video segment to be detected as the target video segment when the block kinetic energy index exceeds a preset threshold.

13. The apparatus according to claim 12, characterized in that, The formula for calculating the block kinetic energy index is: ； in, Let be the block kinetic energy index of the block n to be detected in direction d. Let s and be normal distribution functions. It is the normal distribution constant. This is an adjustment coefficient used to adjust the weight of the block n to be detected. Let S be the relative pixel gradient of the block n to be detected in the direction d, and let S be the pixel size of the block n to be detected.

14. The apparatus according to claim 8, characterized in that, The device further includes: The sending module is used to send the video segment to be detected to an edge-side device or a cloud-side device when it is determined that the video segment to be detected is a target video segment.

15. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

Images