Object state determination method and device, equipment, medium and product
By segmenting and calculating the difference between IoT video frames, abnormal frames are filtered out and the state is determined using a recognition model. This solves the problem of inaccurate state judgment in IoT videos and achieves more efficient and accurate state judgment.
Patent Information
- Application Number
- CN202511960670.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-03-06
AI Technical Summary
In existing technologies, determining the status of protected objects in IoT videos by periodically extracting video frames can easily lead to inaccurate judgment results, with problems such as video frame loss and inaccurate status judgment.
The video frames collected by IoT devices are divided into multiple image blocks according to a preset size. Vectors are calculated based on the center pixel values of the image blocks and the center pixel values of the video frames. Abnormal frames are filtered out by difference and the target state is determined by the recognition model.
It enables rapid filtering of candidate abnormal video frame sequences, improves the accuracy of target protection object status judgment, avoids the heavy burden of directly inputting full data into the model, and enhances the accuracy of status judgment.
Smart Images
Figure CN121617017A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, and in particular relates to a method, apparatus, device, medium and product for determining the state of an object. Background Technology
[0002] Internet of Things (IoT) video is video captured by IoT acquisition devices. Security protection can be achieved through the analysis and processing of IoT video. For example, by analyzing and processing IoT video of important data centers or equipment, security protection can be implemented for those data centers or equipment. Currently, a method of periodically extracting video frames is used to reduce the data volume, and then the content of each extracted video frame is analyzed to determine the status of the protected object.
[0003] However, while periodically extracting video frames can reduce the amount of data processed, it also causes the loss of some video frames. Therefore, there is a problem that the status of the protected object is not accurately determined based on the extracted video frames. Summary of the Invention
[0004] This application provides a method, apparatus, device, medium, and product for determining the state of an object, which can accurately determine the target state of the object to be protected.
[0005] In a first aspect, embodiments of this application provide a method for determining the state of an object, including: Acquire video footage of the monitored target / protected object collected by IoT devices; Each video frame in the video is divided into multiple image blocks according to a preset size; For each video frame, a vector for each image block is determined based on the pixel values of the center pixels of multiple image blocks in the video frame and the pixel value of the center pixel of the video frame. The difference between the vectors of multiple image blocks in each video frame and the vectors of multiple image blocks in the standard image is determined. The difference is used to characterize the degree of anomaly in the video frame. The standard image is a pre-acquired standard image that characterizes the target protected object in a normal state. From the video, at least one video frame with a difference greater than or equal to a preset threshold is identified to obtain the target video frame sequence; Input the target video frame sequence into the preset recognition model to determine the target state of the target protected object. The target state includes normal state or abnormal state.
[0006] Secondly, embodiments of this application provide an apparatus for determining the state of an object, comprising: The acquisition module is used to acquire video of the monitored target protected object collected by IoT devices; The segmentation module is used to divide each video frame in the video according to a preset size to obtain multiple image blocks; The determination module is used to determine the vector of each image block for each video frame based on the pixel values of the center point pixels of multiple image blocks in the video frame and the pixel value of the center pixel of the video frame. The determination module is also used to determine the difference between the vectors of multiple image blocks in each video frame and the vectors of multiple image blocks in the standard image, respectively. The difference is used to characterize the degree of anomaly of the video frame. The standard image is a pre-acquired standard image that characterizes the target protected object in a normal state. The determination module is also used to determine at least one video frame from the video whose difference is greater than or equal to a preset threshold, so as to obtain a target video frame sequence; The determination module is also used to input the target video frame sequence into a preset recognition model to determine the target state of the target protected object, which includes a normal state or an abnormal state.
[0007] Thirdly, embodiments of this application provide an electronic device, the device comprising: Processor and memory storing computer program instructions; The method for determining the state of the object used by the processor to execute the first aspect mentioned above when executing computer program instructions.
[0008] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the method for determining the state of an object as described in the first aspect.
[0009] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when processed by a processor, implements the method for determining the object state described in the first aspect.
[0010] The object state determination method, apparatus, device, medium, and product provided in this application embodiment acquires video of the monitored target protected object collected by IoT devices, divides each video frame in the video according to a preset size to obtain multiple image blocks, constructs a vector of the image block by using the pixel values of the center pixel of the image block and the center pixel of the frame, determines the difference degree based on the vectors of multiple image blocks in each video frame and the vectors of multiple image blocks in a standard image, obtains a target video frame sequence by filtering video frames with a difference degree greater than or equal to a preset value, and identifies the target video frame sequence by a recognition model to obtain the state of the target protected object. This achieves rapid screening of candidate abnormal target video frame sequences, and then the recognition model performs accurate state classification. This avoids the heavy burden of directly inputting all data into the model and can accurately determine the state of the target protected object through recognition, thus improving the accuracy of the state judgment of the target protected object. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart illustrating a method for determining the state of an object, provided in some embodiments of this application.
[0013] Figure 2 This is a schematic diagram of an exemplary video frame provided for some embodiments of this application.
[0014] Figure 3 This is a flowchart illustrating another method for determining the state of an object provided in some embodiments of this application.
[0015] Figure 4 This is a flowchart illustrating yet another method for determining the state of an object, provided in some embodiments of this application.
[0016] Figure 5 This is a flowchart illustrating yet another method for determining the state of an object, provided in some embodiments of this application.
[0017] Figure 6 This is a schematic diagram of an object state determination device provided for some embodiments of this application.
[0018] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0019] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0021] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application first specifically explains the problems existing in the related technologies: Internet of Things (IoT) video refers to surveillance video collected by IoT devices targeting protected objects. Existing technologies, which use frame extraction to determine the state of the protected object, are prone to inaccurate judgments.
[0022] Based on this, embodiments of this application provide a method, apparatus, device, medium, and product for determining the state of an object, which can solve the above-mentioned problems. The method for determining the state of an object provided by embodiments of this application will be described in detail below.
[0023] In some embodiments, such as Figure 1 As shown in the embodiments of this application, a method for determining the state of an object is provided, which may include: S110: Acquire video of the monitored target protected object collected by IoT devices.
[0024] The aforementioned video can be a video captured by an IoT device (such as a camera) targeting a protected object (such as a computer room or equipment).
[0025] S120: Divide each video frame in the video into multiple image blocks according to a preset size.
[0026] Each video frame (i.e., an image) is divided into blocks of a preset size. For example, if the preset size is 100×100 pixels, the video frame can be divided into image blocks of 10×10 pixels, starting from the top left corner. In this way, a high-resolution image is converted into multiple small image blocks.
[0027] S130: For each video frame, determine the vector of each image block based on the pixel values of the center point pixels of multiple image blocks in the video frame and the pixel value of the center pixel of the video frame.
[0028] The RGB value of the center pixel of each image block can be used as the standard value for that block. If the image block does not have a center point, any pixel RGB value can be used, for example, the RGB value of the top left pixel.
[0029] For each image block, the difference between the standard value of the image block and the pixel value of the center pixel of the video frame can be calculated to obtain the vector of the image block, namely (R difference, G difference, B difference). Here, the ratio of the standard value of the image block to the pixel value of the center pixel of the video frame can also be calculated to obtain the vector, namely (R ratio, G ratio, B ratio). S140: Determine the difference between the vectors of multiple image blocks in each video frame and the vectors of multiple image blocks in the standard image. The difference is used to characterize the degree of anomaly in the video frame. The standard image is a pre-acquired standard image that characterizes the target protected object in a normal state.
[0030] Here, the same segmentation scheme as the video frame is used for the standard image to obtain the same number of image blocks at the same positions.
[0031] For image patches corresponding to locations in a video frame and a standard image, the distance between their vectors can be calculated. Distance calculation methods can include Euclidean distance, cosine distance, or Manhattan distance. Aggregating the distances between all image patches in the video frame and their corresponding patches in the standard image yields the dissimilarity score. It's conceivable that a larger dissimilarity score indicates a greater difference between the current video frame and the standard image in a normal state, i.e., a higher degree of anomaly.
[0032] S150: Determine at least one video frame from the video whose difference is greater than or equal to a preset threshold, and obtain the target video frame sequence.
[0033] The calculated difference of each video frame is compared with a preset threshold. If the difference of a video frame is greater than or equal to the threshold, the frame is considered an anomalous frame. All the video frames identified as anomalous are extracted in chronological order to form a target video frame sequence. This sequence represents the time period in the video where anomalies may have occurred.
[0034] S160: Input the target video frame sequence into the preset recognition model to determine the target state of the target protected object. The target state includes normal state or abnormal state.
[0035] The selected target video frame sequence is input into a preset recognition model, such as an artificial intelligence (AI) recognition model. It can be imagined that after training, the recognition model can accurately identify the input abnormal video segments with reference to a standard image, and finally output the status of the target protected object, that is, the normal state or the abnormal state.
[0036] AI recognition models can be neural network models. By feeding them a large number of video clips of various anomalies and standard images, the accuracy of their recognition of anomalies can be trained, thus obtaining a well-trained AI recognition model.
[0037] This application embodiment acquires video of the monitored target object collected by IoT devices, divides each video frame in the video according to a preset size to obtain multiple image blocks, constructs a vector of the image block by using the pixel values of the center pixel of the image block and the center pixel of the frame, and determines the degree of difference based on the vectors of multiple image blocks in each video frame and the vectors of multiple image blocks in a standard image. This allows for fine-grained determination of the degree of abnormality of each video frame. By filtering video frames with a degree of difference greater than or equal to a preset value, a target video frame sequence is obtained. The target video frame sequence is then identified by a recognition model to obtain the state of the target protected object. This enables rapid filtering of candidate abnormal target video frame sequences, followed by accurate state classification by the recognition model. This avoids the heavy burden of directly inputting all data into the model and can accurately determine the state of the target protected object through recognition, thus improving the accuracy of state judgment of the target protected object.
[0038] In some embodiments, before dividing each video frame in the video into multiple image blocks according to a preset size, the determination method further includes: Obtain the video's memory usage, resolution, data processing volume of the IoT device's central processing unit, and the protection level of the target object; determine the preset size based on the memory usage, resolution, data processing volume, and protection level.
[0039] The memory usage of a video refers to the size of the IoT video file to be processed. The larger the memory, the longer the video duration or the higher the resolution, and the greater the processing pressure.
[0040] Video resolution refers to the pixel size of a video frame. The higher the resolution, the more pixel information a single frame contains, and the higher the computational complexity.
[0041] The data processing capacity of an IoT device's central processing unit (CPU) refers to the maximum amount of data (in bits) that the CPU can process in one processing cycle under the current load. For example, a CPU's processing capacity is 512 bits per clock cycle. This parameter reflects the device's real-time processing capability.
[0042] The protection level of the target object is a parameter set by the user or system to characterize the level of security requirements for the object. For example, the protection level can be divided into three levels: "low," "medium," and "high," each corresponding to different levels of security sensitivity.
[0043] Based on the above parameters, the maximum number of initial blocks that the device can cover in one processing run under its current capabilities is first calculated. Based on the video's memory footprint w and resolution, the average size of a single pixel (in bits) can be estimated. For example, if the video's memory footprint w and resolution are 176×144, then the memory footprint per pixel is w / (176×144).
[0044] Then, based on the CPU's data processing capacity, calculate the maximum number of initial blocks that it can process in one processing cycle. For example, if the CPU's data processing capacity is 512 bits, then there can be a maximum of the following number of initial blocks in a video frame: Maximum number of initial blocks = [CPU data processing capacity / memory occupied by a single pixel] (where [] is the floor function). For example, if the CPU's processing capacity is 512 bits and the memory occupied by a single pixel is 1 bit, then the maximum number of initial blocks is 512.
[0045] Next, a scaling factor is determined based on the protection level value. For example, if the protection level is "low," the scaling factor is 1; if it is "medium," the scaling factor is 1 / 2; and if it is "high," the scaling factor is 1 / 3. The maximum initial block count is multiplied by the scaling factor to obtain the adjusted block count. For example, if the maximum initial block count is 900 and the protection level is "high" (scaling factor 1 / 3), the adjusted block count is 300.
[0046] Finally, the adjusted number of blocks is converted into the side length of a square, which is used as the final preset size. For example, the square root of the adjusted number of blocks is taken and rounded to obtain the side length of a square. For instance, if the adjusted number of blocks is 300, its square root is approximately 17.32, which, after rounding, becomes 17.
[0047] The calculated side length is used as the preset size. For example, the preset size is 17×17 pixels, which means that the size of the image block within each video frame is 17×17 pixels.
[0048] This application embodiment obtains the video's memory usage, resolution, CPU data processing volume, and protection strength value, and determines a preset size based on these parameters. The preset size calculated in this way avoids the resource waste or insufficient protection that may be caused by a fixed segmentation size. This allows the image blocks obtained by segmenting video frames based on the preset size to capture minute abnormal changes in the video frames more precisely, thereby improving the accuracy and sensitivity of the protection.
[0049] In some embodiments, based on the pixel values of the center point pixels of multiple image blocks in a video frame and the pixel values of the center pixel of the video frame, the method includes: For any target image block in a video frame, calculate the difference between the pixel value of the center pixel of the video frame and the pixel value of the center pixel of the target image block, and determine the vector of the target image block.
[0050] For any given image patch, the following difference can be calculated: R-difference = R-value of the center pixel of the video frame - R-value of the center pixel of the target image block; G-difference = G-value of the center pixel of the video frame - G-value of the center pixel of the target image block; B-difference = B-value of the center pixel of the video frame - B-value of the center pixel of the target image block; The vector of the target image patch can be represented as: [R difference, G difference, B difference].
[0051] For example, regarding such Figure 2 The vector of image block A in the video frame shown is: (R difference is the R value of image block E - the R value of image block A, G difference is the G value of image block E - the G value of image block A, B difference is the B value of image block E - the B value of image block A).
[0052] In such Figure 2 In the video frame shown, since the center pixel of image block E coincides with the center pixel of the entire video frame, the vector of image block E is (R value of image block E, B value of image block E, G value of image block E).
[0053] This embodiment determines the image block vector by the difference between the pixel value of the center pixel of each video frame and the pixel value of the center pixel of the image block. This vector includes not only the color information of the image block itself but also its relative position within the video frame. It's conceivable that if the video frame experiences a general change in illumination (e.g., the light brightens or darkens), the RGB values of the center pixel of the video frame and the center pixel of the target image block will change simultaneously and proportionally. In this case, the calculated difference will be relatively stable, thus reducing the impact of illumination changes on the feature vector. The vector calculated in this way is unaffected by the environment, which is beneficial for subsequently determining the target state of the protected object.
[0054] In some embodiments, such as Figure 3 As shown, determining the difference between the vectors of multiple image patches in each video frame and the vectors of multiple image patches in a standard image can include: S310: For any target video frame in the video, determine the abrupt change degree of the target video frame based on the vectors of multiple image blocks in the target video frame and the vectors of multiple image blocks in the comparison video frame. The vectors of multiple image blocks in the comparison video frame are determined based on the vectors of multiple image blocks in at least one video frame preceding the target video frame. The abrupt change degree characterizes the continuity between the target video frame and the comparison video frame.
[0055] Here, for any target video frame, at least one preceding video frame can be selected as the comparison video frame. The comparison video frame can be a single video frame or a combination of multiple video frames. When the comparison video frame is a combination of frames, the vector of each image block in the comparison video frame is the average of the vectors of image blocks with the same position in each frame of the frame combination.
[0056] For image patches corresponding to positions in the target and comparison video frames, the distances between their vectors (such as Euclidean distance, cosine distance, etc.) can be calculated. Based on the distances of all image patches, a mutation degree is calculated, which characterizes the degree of continuity between the target and comparison video frames. The smaller the mutation degree, the more similar the two frames are, and the more continuous their content; the larger the mutation degree, the greater the difference between the two frames, indicating a mutation has occurred.
[0057] In some examples, the aforementioned abruptness can be determined based on the average or standard deviation of the distances across all blocks.
[0058] S320: From the video, identify video frames with a mutation rate less than a preset value to obtain a video frame sequence.
[0059] Video frames with a mutation rate less than a preset value are grouped together to form a video frame sequence. This video frame sequence represents a segment of video where the content is continuous and does not undergo drastic changes.
[0060] S330: For each video frame in the video frame sequence, determine the degree of difference based on the vectors of multiple image patches in the video frame and the vectors of multiple image patches in the standard image.
[0061] Here, the distance can be determined based on the vector of each image patch in the video frame and the vector of the corresponding image patch in the standard image, and the degree of difference can be determined based on the mean and standard deviation of multiple distances.
[0062] From the video, at least one video frame with a difference greater than or equal to a preset threshold is identified to obtain a target video frame sequence, including: S340: From the video frame sequence, determine at least one video frame whose difference is greater than or equal to a preset threshold to obtain the target video frame sequence.
[0063] The target video frame sequence can be obtained by determining at least one video frame with a difference greater than or equal to a preset threshold from a video frame sequence formed by multiple video frames with a mutation degree less than or equal to a preset value.
[0064] This application embodiment first forms a video frame sequence based on multiple video frames with a mutation degree less than or equal to a preset value. This mutation degree filtering eliminates a large number of meaningless frames with drastic changes caused by environmental factors, thus achieving data noise reduction. Then, at least one video frame with a difference degree greater than or equal to a preset threshold is determined from the video frame sequence to obtain the target video frame sequence. Based on the above two-stage filtering, a continuous target video frame sequence can be selected, making the target status of the protected object determined based on this target video frame sequence more accurate.
[0065] In some embodiments, such as Figure 4 As shown, the video consists of N video frames. For any target video frame in the video, the abrupt change of the target video frame is determined based on the vectors of multiple image patches in the target video frame and the vectors of multiple image patches in the comparison video frame. This can include: S410: The first video frame in the video is determined as the comparison video frame. For the second video frame in the video, the abruptness of the second video frame is determined based on the vectors of multiple image patches in the second video frame and the vectors of multiple image patches in the first video frame.
[0066] The first video frame (i=1) in the video stream is determined as the initial comparison video frame. For the second video frame (i=2) in the video stream, the distance between the corresponding image block vector of each image block in the second video frame and the current comparison video frame (i.e. the first video frame) can be calculated. Based on the distance of all blocks, the mutation degree of the second video frame is calculated.
[0067] S420: If the abrupt change of the second video frame is less than a preset value, the combination of the first and second video frames is used as the comparison video frame, and the vectors of multiple image blocks in the comparison video frame are determined based on the vectors of multiple image blocks in the first video frame and the vectors of multiple image blocks in the second video frame; or, if the abrupt change of the second video frame is greater than or equal to the preset value, the first video frame is kept unchanged as the comparison video frame.
[0068] If the abrupt change is less than a preset value, it indicates that if the second frame is very similar to the first frame, the content of the second frame is continuous with the first frame. In this case, the first and second frames can be combined into a new comparison video frame. The vector of this new comparison video frame is no longer a single-frame vector, but rather constitutes the average of the block vectors corresponding to all frames within the combination. Or... If the abrupt change is greater than or equal to a preset value, it indicates that the second frame differs significantly from the first frame, signifying a sudden change has occurred. In this case, the first frame can be kept as the comparison video frame, and preparations can be made to process the next frame.
[0069] For i ranging from 3 to N, perform the following steps in sequence: S430: For the i-th video frame, determine the abrupt change degree of the i-th video frame based on multiple image block vectors in the i-th video frame and multiple image block vectors of the current comparison video frame.
[0070] For the i-th video frame, calculate the degree of abrupt change between it and the current comparison video frame (which may be a single frame or a combination of frames).
[0071] S440: Determine whether the abrupt change of the i-th video frame is less than the preset value.
[0072] S450: If the mutation degree of the i-th video frame is less than the preset value, combine the i-th video frame with the current comparison video frame to obtain a new comparison video frame. Determine the vectors of multiple image blocks in the new comparison video frame based on the vectors of multiple image blocks in the i-th video frame and the vectors of multiple image blocks in the current comparison video frame, and continue to calculate the mutation degree of the (i+1)-th video frame.
[0073] If the i-th video frame is similar to the current comparison video frame (i.e., the abrupt change is less than a preset value), then this frame is combined with the current comparison video frame to form a new, longer comparison video frame. The vector of the new comparison frame is also updated to the mean of the block vectors of all frames in the combination.
[0074] S460: If the mutation degree of the i-th video frame is greater than or equal to the preset value, keep the current comparison video frame unchanged and continue to calculate the mutation degree of the (i+1)-th video frame.
[0075] If the i-th video frame is not similar to the current comparison video frame (i.e., the abrupt change is greater than or equal to the preset value), then the current comparison video frame remains unchanged, and the next frame (i+1) is processed. At this point, the current comparison video frame (i.e., the combination of all previous consecutive similar frames) is determined to be a complete video segment.
[0076] It is conceivable that the comparison video frames obtained based on the above processing can be used as the aforementioned video frame sequence, and the target video frame sequence can be obtained by filtering based on this video frame sequence.
[0077] This application embodiment compares the mutation degree of the current video with that of the comparison video frame to finally determine the video frame sequence with a mutation degree less than a preset value. In the comparison process, similar frames are dynamically combined to form comparison video frames and the mean of the combined vector is calculated as the comparison benchmark, which makes the judgment of content mutation more accurate.
[0078] In some embodiments, determining the abrupt change of a target video frame based on vectors of multiple image patches in the target video frame and vectors of multiple image patches in a comparison video frame includes: Calculate the distance between the vector of each image block in the target video frame and the vector of the image block at the same spatial location in the comparison video frame to obtain multiple first distances; determine the abrupt change degree based on the multiple first distances and the average of the first distances.
[0079] For the target video frame and the comparison video frame, the distance between the vector of each image block in the target video frame and the vector of the image block at the same spatial position in the comparison video frame is calculated to obtain multiple first distances. That is, the first distance is calculated between the first image block in the target video frame and the first image block in the comparison video frame, the first distance is calculated between the second image block and the second image block, and so on.
[0080] The degree of mutation can be determined based on the following expression. t : (1) in, Let be the first distance between any image patch and the corresponding image patch in the compared video frame. The first distance is the average value of all the first distances within the target video frame. I The total number of image blocks within the target video frame.
[0081] The embodiments of this application are based on the mutation degree calculated in the above manner. This method not only considers the average level of distance, but more importantly, it considers the distribution and fluctuation of distance, so as to accurately determine the mutation degree of video frames.
[0082] In some embodiments, such as Figure 5 As shown, determining the difference between the vectors of multiple image patches in each video frame and the vectors of multiple image patches in a standard image can include: S510: Calculate the distance between the vector of each image block in the video frame and the vector of the image block at the same spatial location in the standard image to obtain multiple second distances.
[0083] For each image patch in a video frame, the distance between it and the corresponding image patch vector in the standard image can be calculated.
[0084] S520: Determine the distribution value and fluctuation value based on the mean and standard deviation of multiple second distances, respectively.
[0085] The distribution value f can be determined based on the following expression: (2) The fluctuation value b is determined based on the following expression: (3) in, Second distance, The second distance is the average value of all distances within that video frame. Standard deviation U The total number of image blocks within the video frame.
[0086] S530: In a video frame, determine the number of first image blocks in which both the distribution value and the fluctuation value are within the corresponding preset range, and determine the number of second image blocks in which at least one of the distribution value and the fluctuation value is not within the corresponding preset range.
[0087] For each image block in a video frame, determine whether its distribution value and fluctuation value are both within their respective preset ranges. If both the distribution value and fluctuation value are within the preset range, the image block is determined to be the first image block. If at least one of the distribution value or fluctuation value is outside the preset range, the image block is determined to be the second image block.
[0088] The preset range corresponding to the distribution value is The preset range corresponding to the fluctuation value is .
[0089] Here, the above This can be the distribution value of the first block in the top left corner of the standard image. This can be the fluctuation value of the first block in the top left corner of the standard image. S540: Determine the degree of difference based on the number of the first image patch and the number of the second image patch.
[0090] The difference can be calculated based on the following expression: Difference = Number of second image patches / (Number of first image patches + Number of second image patches). The greater the difference, the more anomalous areas there are between the video frame and the standard image, and the higher the degree of anomalousness.
[0091] Here, the preset threshold corresponding to the degree of difference can be determined based on the protection strength value of the target object. It can be imagined that different protection strength values correspond to different preset thresholds.
[0092] In some examples, the preset threshold can be determined based on the following expression: The preset threshold is set to softmax{0.5(number of pixels in an image block / total number of pixels in a single video frame) x}, where x is the protection level.
[0093] This application embodiment obtains multiple second distances by calculating the distance between the vector of each image block in a video frame and the vector of an image block at the same spatial location in a standard image. Based on the average and standard deviation of these second distances, distribution values and fluctuation values are determined. First and second image blocks in the video frame are then selected based on these distribution values and fluctuation values. The degree of difference is determined based on the number of first and second image blocks. By analyzing the distribution and fluctuation values, transient, random noise interference can be filtered out, responding only to continuous, statistically regular abnormal changes. This reduces the false alarm rate, improves the accuracy of anomaly detection, and makes the determination of the target state of the protected object more accurate in complex environments.
[0094] Based on the same inventive concept, embodiments of this application also provide an object state determination device.
[0095] In some embodiments, such as Figure 6 As shown, this application embodiment provides an object state determination device, including: The acquisition module 601 is used to acquire video of the monitored target protected object collected by the Internet of Things device; The segmentation module 602 is used to segment each video frame in the video according to a preset size to obtain multiple image blocks; The determination module 603 is used to determine the vector of each image block for each video frame based on the pixel values of the center point pixels of multiple image blocks in the video frame and the pixel values of the center pixel of the video frame. The determination module 603 is also used to determine the difference degree between the vectors of multiple image blocks in each video frame and the vectors of multiple image blocks in the standard image, respectively. The difference degree is used to characterize the degree of abnormality of the video frame. The standard image is a pre-acquired standard image that characterizes the target protected object in a normal state. The determining module 603 is further configured to determine at least one video frame from the video whose difference is greater than or equal to a preset threshold, thereby obtaining a target video frame sequence; The determination module 603 is also used to input the target video frame sequence into a preset recognition model to determine the target state of the target protected object, including normal state or abnormal state.
[0096] This application embodiment acquires video of the monitored target object collected by an IoT device through an acquisition module. A segmentation module divides each video frame in the video into multiple image blocks according to a preset size. A determination module constructs a vector for each image block using the pixel values of the center pixel of the image block and the center pixel of the frame. Based on the vectors of multiple image blocks in each video frame and the vectors of multiple image blocks in a standard image, the difference is determined. By filtering video frames with a difference greater than or equal to a preset value, a target video frame sequence is obtained. A recognition model identifies the target video frame sequence to determine the state of the target protected object. This allows for rapid filtering of candidate abnormal target video frame sequences, followed by precise state classification by the recognition model, accurately determining the state of the target protected object and improving the accuracy of state judgment.
[0097] In some embodiments, the segmentation module is configured to: Obtain the video's memory usage, resolution, the data processing volume of the IoT device's central processing unit, and the protection level of the target object; The preset size is determined based on memory usage, resolution, data processing volume, and protection level.
[0098] In some embodiments, the determining module is configured to: For any target image block in a video frame, calculate the difference between the pixel value of the center pixel of the video frame and the pixel value of the center pixel of the target image block, and determine the vector of the target image block.
[0099] In some embodiments, the determining module is configured to: For any target video frame in the video, the abruptness of the target video frame is determined based on the vectors of multiple image patches in the target video frame and the vectors of multiple image patches in the comparison video frame. The vectors of multiple image patches in the comparison video frame are determined based on the vectors of multiple image patches in at least one video frame preceding the target video frame. The abruptness characterizes the degree of continuity between the target video frame and the comparison video frame. From the video, identify video frames with a mutation rate less than a preset value to obtain a video frame sequence; For each video frame in the video frame sequence, the degree of difference is determined based on the vectors of multiple image patches in the video frame and the vectors of multiple image patches in the standard image; From the video, at least one video frame with a difference greater than or equal to a preset threshold is identified to obtain a target video frame sequence, including: From the video frame sequence, at least one video frame with a difference greater than or equal to a preset threshold is determined to obtain the target video frame sequence.
[0100] In some embodiments, the video comprises N video frames, and the determining module is configured to: The first video frame in the video is identified as the comparison video frame. For the second video frame, the abrupt change of the second video frame is determined based on the vectors of multiple image patches in the second video frame and the vectors of multiple image patches in the first video frame. If the abrupt change of the second video frame is less than a preset value, the combination of the first and second video frames is used as the comparison video frame, and the vectors of multiple image blocks in the comparison video frame are determined based on the vectors of multiple image blocks in the first video frame and the vectors of multiple image blocks in the second video frame; or, if the abrupt change of the second video frame is greater than or equal to the preset value, the first video frame is kept unchanged as the comparison video frame. For i ranging from 3 to N, perform the following steps in sequence: For the i-th video frame, the abrupt change degree of the i-th video frame is determined based on multiple image patch vectors in the i-th video frame and multiple image patch vectors of the current comparison video frame. Determine whether the abrupt change of the i-th video frame is less than a preset value; If the abrupt change in the i-th video frame is less than a preset value, the i-th video frame is combined with the current comparison video frame to obtain a new comparison video frame. Based on the vectors of multiple image patches in the i-th video frame and the vectors of multiple image patches in the current comparison video frame, the vectors of multiple image patches in the new comparison video frame are determined, and the abrupt change in the (i+1)-th video frame is calculated; or If the mutation degree of the i-th video frame is greater than or equal to the preset value, keep the current comparison video frame unchanged and continue to calculate the mutation degree of the (i+1)-th video frame.
[0101] In some embodiments, the determining module is configured to: Calculate the distance between the vector of each image block in the target video frame and the vector of the image block at the same spatial location in the comparison video frame to obtain multiple first distances; The degree of mutation is determined based on multiple first distances and the average of the first distances.
[0102] In some embodiments, the determining module is configured to: Calculate the distance between the vector of each image patch in the video frame and the vector of the image patch at the same spatial location in the standard image to obtain multiple second distances; Based on the mean and standard deviation of multiple second distances, the distribution value and fluctuation value are determined respectively; In a video frame, determine the number of first image blocks in which both the distribution value and the fluctuation value are within the corresponding preset range, and determine the number of second image blocks in which at least one of the distribution value and the fluctuation value is not within the corresponding preset range; The degree of difference is determined based on the number of the first image patch and the number of the second image patch.
[0103] The apparatus of the above embodiments is used to implement the method for determining the state of the corresponding object in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0104] Figure 7 A schematic diagram of the hardware structure of an electronic device is provided in the application embodiment.
[0105] The electronic device 700 may include a processor 701 and a memory 702 storing computer program instructions.
[0106] Specifically, the processor 701 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0107] Memory 702 may include mass storage for data or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 702 may include removable or non-removable (or fixed) media. Where appropriate, memory 702 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 702 is non-volatile solid-state memory.
[0108] In a particular embodiment, memory 702 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.
[0109] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this application.
[0110] The processor 701 reads and executes computer program instructions stored in the memory 702 to implement any of the object state determination methods in the above embodiments.
[0111] In one example, the electronic device may also include a communication interface 703 and a bus 704. Wherein, as... Figure 7 The processor 701, memory 702, and communication interface 703 are connected through bus 704 and complete communication with each other.
[0112] The communication interface 703 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0113] Bus 704 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 704 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0114] The electronic devices described above are used to implement the corresponding object state determination method in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0115] Furthermore, in conjunction with the object state determination method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the object state determination methods in the above embodiments.
[0116] Furthermore, in conjunction with the object state determination method in the above embodiments, this application embodiment can provide a computer program product for implementation. When the instructions of this computer program product are executed by the processor of an electronic device, they implement any of the object state determination methods in the above embodiments.
[0117] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.
[0118] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0119] It should also be noted that the exemplary embodiments mentioned in this application describe methods or apparatuses based on a series of steps or devices. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0120] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.
[0121] The above description is merely a specific embodiment of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method of determining a state of an object, characterized by, The method comprises: acquiring a video of a monitoring target protection object collected by an Internet of Things device; segmenting each video frame in the video according to a preset size to obtain a plurality of image blocks; for each video frame, determining a vector of each image block based on a pixel value of a center pixel of the image block and a pixel value of a center pixel of the video frame; determining a difference degree of the vector of each image block in each video frame and the vector of a plurality of image blocks in a standard image, the difference degree being used to represent an abnormality degree of the video frame, wherein the standard image is a standard image representing a normal state of the target protection object and is acquired in advance; from the video, determining at least one video frame with a difference degree greater than or equal to a preset threshold to obtain a target video frame sequence; inputting the target video frame sequence into a preset recognition model to determine a target state of the target protection object, the target state including a normal state or an abnormal state.
2. The method of determining the state of an object according to claim 1, wherein, Before the segmenting each video frame in the video according to a preset size to obtain a plurality of image blocks, the method further comprises: acquiring an occupied memory, a resolution of the video, a data processing amount of a central processing unit of the Internet of Things device, and a protection intensity value of the target protection object; determining the preset size based on the occupied memory, the resolution, the data processing amount, and the protection intensity value.
3. The method of claim 1, wherein The determining a vector of each image block based on a pixel value of a center pixel of the image block and a pixel value of a center pixel of the video frame comprises: for any target image block in the video frame, calculating a difference value between the pixel value of the center pixel of the video frame and the pixel value of the center pixel of the target image block to determine the vector of the target image block.
4. The method of claim 1, wherein The determining a difference degree of the vector of each image block in each video frame and the vector of a plurality of image blocks in a standard image comprises: for any target video frame in the video, determining a mutation degree of the target video frame based on the vector of a plurality of image blocks in the target video frame and the vector of a plurality of image blocks in a comparison video frame, wherein the vector of the plurality of image blocks in the comparison video frame is determined based on the vector of a plurality of image blocks in at least one video frame before the target video frame, and the mutation degree represents a continuity degree of the target video frame and the comparison video frame; from the video, determining a video frame with a mutation degree less than a preset value to obtain a video frame sequence; for each video frame in the video frame sequence, determining the difference degree based on the vector of a plurality of image blocks in the video frame and the vector of a plurality of image blocks in a standard image; The determining at least one video frame with a difference degree greater than or equal to a preset threshold from the video to obtain a target video frame sequence comprises: from the video frame sequence, determining at least one video frame with a difference degree greater than or equal to a preset threshold to obtain the target video frame sequence.
5. The method of determining the state of an object according to claim 4, wherein, The video comprises N video frames, and the mutation degree of any target video frame in the video is determined based on vectors of image blocks in the target video frame and vectors of image blocks in a comparison video frame, comprising: a first video frame in the video is determined as the comparison video frame, and a mutation degree of a second video frame in the video is determined based on vectors of image blocks in the second video frame and vectors of image blocks in the first video frame; if the mutation degree of the second video frame is less than the preset value, a combination of the first video frame and the second video frame is taken as the comparison video frame, and vectors of image blocks in the comparison video frame are determined based on vectors of image blocks in the first video frame and vectors of image blocks in the second video frame; or if the mutation degree of the second video frame is greater than or equal to the preset value, the first video frame is kept unchanged as the comparison video frame; for i from 3 to N, the following steps are sequentially executed: for the i-th video frame, a mutation degree of the i-th video frame is determined based on vectors of image blocks in the i-th video frame and vectors of image blocks in the current comparison video frame; it is judged whether the mutation degree of the i-th video frame is less than the preset value; if the mutation degree of the i-th video frame is less than the preset value, the i-th video frame and the current comparison video frame are combined to obtain a new comparison video frame, vectors of image blocks in the new comparison video frame are determined based on vectors of image blocks in the i-th video frame and vectors of image blocks in the current comparison video frame, and the mutation degree of the i+1-th video frame is continued to be calculated; or if the mutation degree of the i-th video frame is greater than or equal to the preset value, the current comparison video frame is kept unchanged, and the mutation degree of the i+1-th video frame is continued to be calculated.
6. The method of claim 4, wherein The mutation degree of the target video frame is determined based on vectors of image blocks in the target video frame and vectors of image blocks in a comparison video frame, comprising: a distance between a vector of each image block in the target video frame and a vector of an image block at the same spatial position in the comparison video frame is calculated to obtain a plurality of first distances; the mutation degree is determined based on the plurality of first distances and an average value of the first distances.
7. The method of claim 1, wherein The difference degree of the vectors of the image blocks in each of the video frames and the vectors of the image blocks in the standard image is respectively determined, comprising: a distance between a vector of each image block in the video frame and a vector of an image block at the same spatial position in the standard image is calculated to obtain a plurality of second distances; a distribution value and a fluctuation value are respectively determined based on an average value and a standard deviation of the plurality of second distances; In the video frame, a quantity of first image blocks in which the distribution value and the fluctuation value are both within corresponding preset ranges is determined, and a quantity of second image blocks in which at least one of the distribution value and the fluctuation value is not within a corresponding preset range is determined; The difference degree is determined based on the quantity of first image blocks and the quantity of second image blocks.
8. An apparatus for determining a state of an object, characterized by Comprise: An acquisition module is configured to acquire a video of a monitoring target protection object collected by an Internet of Things device; A segmentation module is configured to segment each video frame in the video according to a preset size to obtain a plurality of image blocks; A determination module is configured to, for each video frame, determine a vector of each image block based on pixel values of center point pixels of the plurality of image blocks in the video frame and a pixel value of a center pixel point of the video frame; The determination module is further configured to determine a difference degree of the vectors of the plurality of image blocks in each video frame and vectors of a plurality of image blocks in a standard image, respectively, the difference degree being used to represent an abnormality degree of a video frame, wherein the standard image is a standard image representing that the target protection object is in a normal state and is acquired in advance; The determination module is further configured to determine at least one video frame in which the difference degree is greater than or equal to a preset threshold from the video to obtain a target video frame sequence; The determination module is further configured to input the target video frame sequence into a preset recognition model to determine a target state of the target protection object, the target state including a normal state or an abnormal state.
9. An electronic device, comprising: The device comprises a processor and a memory having computer program instructions stored therein; The processor reads and executes the computer program instructions to implement the object state determination method according to any one of claims 1 to 7.
10. A readable storage medium, characterized by, The readable storage medium has computer program instructions stored thereon, and the computer program instructions are executed by a processor to implement the object state determination method according to any one of claims 1 to 7.
11. A computer program product comprising a computer program, characterized in that, The computer program is processed by a processor to implement the object state determination method according to any one of claims 1 to 7.