Target object detection method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a target object detection method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: acquiring an undetected video frame sequence, a fault frame feature data sequence and a detected video frame sequence; screening and generating a historical frame sequence from the detected video frame sequence; performing optimization and duplicate removal processing on the undetected video frame sequence to obtain a to-be-detected video frame sequence; performing the following steps on the to-be-detected video frame sequence: performing target object detection processing on the to-be-detected video frames to obtain a target object information set as a to-be-corrected target object information set; performing pre-correction processing on the to-be-corrected target object information set to obtain a pre-corrected information set; and performing discrimination optimization processing on the pre-correction information set to obtain detection information of each optimized target object, and outputting and displaying the detection information. According to the embodiment, the accuracy of target object detection is improved, and waste of computing resources is reduced.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically to target object detection methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] With the rapid development of artificial intelligence, object detection, as a core task of computer vision, has also seen rapid advancements and is widely used in various fields such as robot navigation, intelligent monitoring, and industrial inspection. Object detection is a technology that identifies, classifies, and locates specific objects contained in images or videos. Currently, the common approach to object detection in images or videos is to perform pixel-level analysis on the directly acquired image or video stream data, and then use a single deep learning model (such as YOLO, Faster R-CNN, etc.) to identify and define all objects of interest and their categories.

[0003] However, when using the above methods to perform target detection on images or video footage, the following technical problems often arise: Pixel-level analysis is performed on the video stream data directly acquired. A single deep learning model is used to identify and frame all objects of interest and their categories in the video. However, the training data and optimization objectives of a single deep learning model are usually designed for specific scenes (such as daytime, sunny days, and urban roads). When the scene changes significantly (such as sudden changes in lighting, inclement weather, or complex backgrounds), the accuracy of object recognition is poor. Furthermore, before performing pixel-level analysis on the video stream data, multiple video frames with high similarity and no obvious differences are not optimized and deduplicated. Since the similarity of multiple consecutive frames is very high, repeatedly processing these almost identical frames during pixel-level analysis wastes computational resources.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide target object detection methods, apparatuses, electronic devices, and computer-readable media to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a target object detection method, which includes: acquiring an undetected video frame sequence; acquiring a preset fault frame feature data sequence and a detected video frame sequence; selecting at least one detected video frame from the detected video frame sequence to generate a historical frame sequence; optimizing and deduplicating the undetected video frame sequence based on the fault frame feature data sequence and the historical frame sequence to obtain a processed undetected video frame sequence as a video frame sequence to be detected; for each video frame to be detected in the video frame sequence to be detected, performing the following steps: performing target object detection processing on the video frame to be detected to obtain a target object information set as a target object information set to be corrected; performing pre-correction processing on each target object information in the target object information set to be corrected based on the video frame to be detected to obtain pre-correction information; determining each pre-correction information as a pre-correction information set; performing discrimination optimization processing on each pre-correction information in the pre-correction information set based on the video frame to be detected to obtain optimized target object detection information; and outputting and displaying the obtained optimized target object detection information.

[0008] Secondly, some embodiments of this disclosure provide a target object detection apparatus, the apparatus comprising: a first acquisition unit configured to acquire an undetected video frame sequence; a second acquisition unit configured to acquire a preset fault frame feature data sequence and a detected video frame sequence; a generation unit configured to filter at least one detected video frame from the detected video frame sequence to generate a historical frame sequence; a first processing unit configured to optimize and deduplicate the undetected video frame sequence based on the fault frame feature data sequence and the historical frame sequence to obtain a processed undetected video frame sequence as a video frame sequence to be detected; and a second processing unit configured to process the above... For each video frame in the video frame sequence to be detected, the following steps are performed: Target object detection processing is performed on the video frame to be detected to obtain a target object information set as a target object information set to be corrected; based on the video frame to be detected, pre-correction processing is performed on each target object information in the target object information set to be corrected to obtain pre-correction information; each pre-correction information is determined as a pre-correction information set; based on the video frame to be detected, discrimination optimization processing is performed on each pre-correction information in the pre-correction information set to obtain optimized target object detection information; an output unit is configured to output and display the obtained optimized target object detection information.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above embodiments of this disclosure have the following beneficial effects: the target object detection method of some embodiments of this disclosure improves the accuracy of target object recognition and reduces the waste of computing resources. Specifically, the reason for the poor accuracy of object recognition and the waste of computing resources is that: pixel-level analysis is performed on the video represented by the directly acquired video stream data, and a single deep learning model is used to identify and frame all target objects of interest and their categories in the video. The training data and optimization targets of the single deep learning model are usually for specific scenes (such as daytime, sunny days, urban roads). When the scene changes significantly (such as sudden changes in lighting, severe weather, complex background), the accuracy of object recognition is poor. At the same time, before performing pixel-level analysis on the video stream data, multiple video frames with high similarity and no obvious difference in the video are not optimized and deduplicated. The similarity of multiple consecutive frames is very high. When performing pixel-level analysis, repeatedly processing these almost identical frames will waste computing resources. Based on this, the target object detection method of some embodiments of this disclosure first obtains the sequence of undetected video frames. Then, it obtains the preset fault frame feature data sequence and the detected video frame sequence. Therefore, a fault frame feature data sequence can be obtained for subsequent removal of problematic video frames from the undetected video frames, and a sequence of detected video frames can be obtained for filtering historical frames. Then, at least one detected video frame is filtered from the above-mentioned sequence of detected video frames to generate a historical frame sequence. This yields a historical frame sequence for removing undetected video frames from the undetected video frame sequence that are similar to historical frames. Then, based on the fault frame feature data sequence and the above-mentioned historical frame sequence, the undetected video frame sequence is optimized and deduplicated to obtain a processed undetected video frame sequence as the video frame sequence to be detected. This yields a video frame sequence to be detected that is free of faults and significantly different from historical frames, reducing resource waste caused by subsequent target object detection on multiple similar frames. Afterwards, for each video frame to be detected in the above-mentioned video frame sequence, the following steps are performed: First, target object detection processing is performed on the above-mentioned video frame to be detected to obtain a target object information set as the target object information set to be corrected. Therefore, after one target detection process, the video frame to be detected yields a set of target object information, including information about each target object in the video frame. This set serves as the target object information set to be corrected for subsequent correction processing. Next, based on the video frame to be detected, each target object information in the target object information set is pre-corrected to obtain pre-corrected information. This results in more accurate pre-corrected information after one correction. Then, each pre-corrected information is defined as a pre-corrected information set. Finally, based on the video frame to be detected, each pre-corrected information in the pre-corrected information set is subjected to discrimination and optimization processing to obtain optimized target object detection information.Therefore, optimized target object detection information after discriminative optimization processing can be obtained. Finally, the obtained optimized target object detection information is output and displayed. This yields a high-accuracy detection result after discriminative optimization processing. This is because by optimizing and deduplicating undetected video frames, multiple video frames with high similarity and no obvious differences or with malfunctions are deleted from the undetected video frames, resulting in a sequence of video frames to be detected. Then, the target object information set obtained from target object detection processing of each video frame in the sequence is pre-corrected, and the target object information in the pre-correction information set is corrected to obtain more accurate target object information as pre-correction information. Finally, combining the images in the video frames to be detected with the pre-correction information set, pre-correction processing and discriminative optimization processing are performed to obtain optimized target object detection information with improved detection accuracy. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the target object detection method according to the present disclosure; Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the target object detection device according to this disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flow 100 of some embodiments of a target object detection method according to the present disclosure is shown. The target object detection method includes the following steps: Step 101: Obtain the sequence of undetected video frames.

[0021] In some embodiments, the execution entity of the target object detection method (e.g., a computing device) can acquire an undetected video frame sequence via a wired or wireless connection. This undetected video frame sequence can be composed of video frames from the undetected video arranged in chronological order. The undetected video can be the video to be detected. The undetected video frames can be the video frames to be detected. It should be noted that the video frames to be detected are added as the last position of the detected video frame sequence after target detection. In practice, the execution entity can acquire the video frame sequence to be detected captured by a camera as the undetected video frame sequence.

[0022] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future wireless connection methods.

[0023] Step 102: Obtain the preset fault frame feature data sequence and the detected video frame sequence.

[0024] In some embodiments, the execution entity can acquire a preset fault frame feature data sequence and a detected video frame sequence. In practice, the execution entity can acquire a pre-set fault frame feature data sequence and a detected video frame sequence from a storage device. The fault frame feature data sequence can be composed of pre-prepared fault frame feature data sorted according to a preset rule. The fault frame feature data can be feature data obtained by feature extraction processing of some preset fault frames. The preset fault frames can be video frames displaying snowflakes, diagonal lines, and a black screen. The detected video frame sequence can be composed of detected video frames arranged in chronological order. The preset rule can be an order of first displaying black screen features, then snowflake features, and finally diagonal line features. The storage device can include, but is not limited to, at least one of the following: a traditional hard disk drive (HDD), a SATA solid-state drive (SATA SSD), and an NVMe M.2 solid-state drive. The feature data can include, but is not limited to, at least one of the following: a color histogram, an oriented gradient histogram, and a feature vector output from feature extraction based on a deep learning model (e.g., a convolutional neural network CNN).

[0025] Step 103: Select at least one detected video frame from the detected video frame sequence to generate a historical frame sequence.

[0026] In some embodiments, the execution entity may select at least one detected video frame from the detected video frame sequence to generate a historical frame sequence. The historical frame sequence may consist of one or more selected detected video frames arranged in chronological order.

[0027] In some optional implementations of certain embodiments, the aforementioned execution entity can filter historical frame sequences from the detected video frame sequences through the following steps: The first step is to use the timestamp corresponding to the first undetected video frame in the aforementioned undetected video frame sequence as the starting timestamp. In practice, the executing entity can first query the timestamp corresponding to the first undetected video frame in the undetected video frame sequence from the storage device, and then determine the timestamp corresponding to the first undetected video frame as the starting timestamp.

[0028] The second step is to generate a historical time interval based on the preset time length and the aforementioned starting timestamp. In practice, the executing entity can extract a time period of the preset time length from the time represented by the aforementioned starting timestamp, and determine the resulting time period as the historical time interval. As an example, the aforementioned starting timestamp could be 11:15, and the preset time length could be 10 minutes, then the generated historical time interval would be from 11:05 to 11:15.

[0029] The third step is to obtain the timestamps corresponding to each detected video frame in the above-mentioned detected video frame sequence as the timestamps of each detected video frame. In practice, the execution entity can obtain the timestamps corresponding to each detected video frame from the storage device.

[0030] The fourth step involves generating a sequence of detected video frame timestamps based on the timestamps of each detected video frame. This sequence can be composed of the timestamps of the detected video frames arranged chronologically. Each timestamp in the sequence corresponds to a detected video frame in the sequence.

[0031] The fifth step is to select at least one timestamp of a detected video frame from the aforementioned sequence of detected video frame timestamps that falls within the aforementioned historical time interval. In practice, the executing entity selects at least one timestamp of a detected video frame from the sequence of detected video frame timestamps whose timestamps represent times within the historical time interval.

[0032] Step 6: Select at least one detected video frame corresponding to the timestamp of at least one detected video frame from the above-mentioned detected video frame sequence as at least one historical frame. In practice, the execution entity may first select at least one detected video frame corresponding to the timestamp of at least one detected video frame from the detected video frame sequence. Then, it may select at least one detected video frame as at least one historical frame.

[0033] Step 7: Add at least one historical frame to a preset historical frame sequence, wherein the existence time of each preset historical frame in the preset historical frame sequence is a preset duration. The preset duration can be a pre-set existence time of the historical frame in the historical frame sequence. In practice, the executing entity can add at least one historical frame to the preset historical frame sequence in chronological order. As an example, the preset duration is 3 minutes, and the preset historical frame sequence is {historical frame 1, historical frame 2, historical frame 3}. The timestamp corresponding to historical frame 1 is 11:05 and has existed in the historical frame for two minutes. The timestamp corresponding to historical frame 2 is 11:07 and has existed in the historical frame for one minute and fifty seconds. The timestamp corresponding to historical frame 3 is 11:08 and has existed in the historical frame for one minute and thirty seconds. The above at least one historical frame includes historical frame A and historical frame B. The timestamp corresponding to historical frame A is 11:11, and the timestamp corresponding to historical frame B is 11:14. Then, the preset historical frame sequence after adding at least one historical frame is {historical frame 1, historical frame 2, historical frame 3, historical frame A, historical frame B}, and the historical frame sequence after one minute is {historical frame 2, historical frame 3, historical frame A, historical frame B}.

[0034] Step 8: Determine the preset historical frame sequence after adding at least one historical frame as the historical frame sequence.

[0035] In addressing the technical problems mentioned above by adopting technical solutions, the application scenario—video scene abrupt changes—often presents the following technical issues: when a scene abruptly changes, the reference value of historical frames is greatly reduced. Continuing to optimize and deduplicate based on historical frames leads to wasted computational resources and reduced detection efficiency. Considering the following requirements for this application scenario: recognizing scene abrupt changes and avoiding the use of meaningless historical frames, we have decided to adopt the following solution: Optionally, after filtering at least one detected video frame from the detected video frame sequence to generate a historical frame sequence, the method may further include: The first step is to select a predetermined number of undetected video frames from the aforementioned undetected video frame sequence in ascending order as the predetermined number of current scene video frames. In practice, the execution entity can select a predetermined number of undetected video frames from the aforementioned undetected video frame sequence in ascending order as the current scene frames, thus obtaining the predetermined number of current scene frames.

[0036] The second step is to generate a sequence of current scene video frames based on the aforementioned preset number of current scene video frames. This sequence can be composed of the preset number of current scene video frames arranged in chronological order.

[0037] The third step involves feature extraction processing on each current scene video frame in the aforementioned current scene video frame sequence to obtain a current scene video frame feature vector sequence. Each current scene video frame feature vector in this sequence corresponds to a current scene video frame within the sequence. In practice, the execution entity can perform HOG (Histogram of Oriented Gradient) feature extraction on each current scene video frame in the aforementioned current scene video frame sequence to obtain feature vectors representing the distribution of each fixed-length target in the image, which are then used as the feature vectors for each current scene video frame.

[0038] The fourth step involves feature extraction processing on each historical frame in the aforementioned historical frame sequence to obtain a sequence of historical frame feature vectors. Each historical frame feature vector in this sequence corresponds to a specific historical frame within the sequence. In practice, the executing entity can perform HOG (Histogram of Oriented Gradient) feature extraction on each historical frame in the sequence to obtain fixed-length feature vectors representing the distribution characteristics of the target object in the image and the target object's features. Then, the executing entity arranges these historical feature vectors according to the order of the corresponding historical frames in the historical frame sequence to obtain the historical feature vector sequence. The target object features can be attributes such as edges, textures, colors, and categories of the target object in the historical frames. As an example, the historical frame sequence can be {historical frame 1, historical frame 2, historical frame 3}. Feature extraction is performed on each historical frame to obtain historical feature vector 1 corresponding to historical frame 1, historical feature vector 2 corresponding to historical frame 2, and historical feature vector 3 corresponding to historical frame 3. The resulting historical feature vector sequence is {historical feature vector 1, historical feature vector 2, historical feature vector 3}.

[0039] The fifth step involves generating a dynamic time warping distance based on the current scene video frame feature vector sequence and the historical frame feature vector sequence. In practice, the execution entity can use the Dynamic Time Warping (DTW) algorithm to calculate the dynamic time warping distance between the current scene video frame feature vector sequence and the historical frame feature vector sequence. The dynamic time warping distance is a numerical value representing the similarity between two sequences, and can be the lower right corner value of the cumulative distance matrix obtained in the dynamic time warping algorithm. For example, sequence A can be {1, 3, 4, 5}, and sequence B can be {2, 3, 4}. Using a recursive formula, the first element from the left of the first row of the cumulative distance matrix is... The second element of the first row is The third element is The first element from the left in the second row from the top is The second element is The third element is And so on, the cumulative distance matrix C that can be obtained is The element 2 in the lower right corner of the cumulative distance matrix C is the dynamic time-warped distance. The above recursive formula can be used... Represented as .in The nth in the cumulative distance matrix Line number The element at the column position. Let represent the i-th element in sequence A. Let represent the i-th element in sequence B. element and The values ​​all start from 0. Represents the local cost, and represents the distance between two elements in sequences A and B (usually using Euclidean distance). For example, Represents the first sequence of A The element and the first element in sequence B The distance between elements.

[0040] Step 6: In response to determining that the aforementioned dynamic time warping distance is greater than a preset dynamic time warping distance, the aforementioned historical frame sequence is set to empty, and an alarm message is generated. The aforementioned preset dynamic time warping distance can be a pre-set threshold value, such as 6, 8, or 10. In practice, when the executing entity determines that the aforementioned dynamic time warping distance is greater than the preset dynamic time warping distance, the executing entity sets the aforementioned historical frame sequence to empty, so that there are no historical frames in the aforementioned historical frame sequence. Then, the preset display content is displayed as the alarm message. As an example, when applied to a real-time detection task of images captured by a camera, the cause of a sudden scene change may be that the camera is shifted due to external force or the camera is obstructed. The preset display content could be: "Warning: Camera position may shift" or "Warning: Camera may be obstructed".

[0041] The above technical solution and its related content, combined with the content of step 104, serve as an inventive point of this disclosure, solving the technical problem of "after a sudden scene change, optimization and deduplication are still performed based on historical frame sequences and video frame sequences to be detected with low reference value, resulting in wasted computational resources and reduced detection efficiency." Factors leading to wasted computational resources and reduced detection efficiency are often as follows: when a scene changes abruptly, the reference value of historical frames is greatly weakened, and continuing to match historical frames leads to wasted computational resources and reduced detection efficiency. Solving these factors can reduce wasted computational resources and improve detection efficiency. To achieve this effect, firstly, a preset number of undetected video frames are selected in ascending order from the above undetected video frame sequence as a preset number of current scene video frames. This yields a preset number of current scene video frames representing the scene in the current detection video. Next, based on the preset number of current scene video frames, a current scene video frame sequence is generated. This yields a current scene video frame sequence representing the current scene. Then, feature extraction processing is performed on each current scene video frame in the above current scene video frame sequence to obtain a current scene video frame feature vector sequence. Therefore, the current scene video frame feature vector sequence used to generate the dynamic time warping distance can be obtained. Then, feature extraction processing is performed on each historical frame in the aforementioned historical frame sequence to obtain a historical frame feature vector sequence. This also yields a historical frame feature vector sequence used to generate the dynamic time warping distance. Then, based on the current scene video frame feature vector sequence and the aforementioned historical frame feature vector sequence, the dynamic time warping distance is generated. This provides a dynamic time warping distance representing the similarity between the current scene video frame sequence and the historical frame feature vector sequence. Finally, in response to determining that the dynamic time warping distance is greater than a preset dynamic time warping distance, the aforementioned historical frame sequence is set to empty, and an alarm message is generated. Thus, when the scene changes drastically, the historical frame sequence can be set to empty and an alarm message can be displayed. Combined with the content of step 104, when the scene changes abruptly, the historical frame sequence is set to empty, and the generation of similarity values ​​between the video frame to be encoded and each historical frame, which has less reference value, is not performed, reducing the waste of computational resources and improving detection efficiency.

[0042] Step 104: Based on the fault frame feature data sequence and the historical frame sequence, optimize and deduplicate the undetected video frame sequence to obtain the processed undetected video frame sequence as the video frame sequence to be detected.

[0043] In some embodiments, the execution entity may optimize and deduplicate the undetected video frame sequence based on the fault frame feature data sequence and the historical frame sequence to obtain the processed undetected video frame sequence as the video frame sequence to be detected.

[0044] In some optional implementations of certain embodiments, the execution entity may optimize and deduplicate the undetected video frame sequence through the following steps to obtain the processed undetected video frame sequence as the video frame sequence to be detected: The first step is to perform the following steps for each undetected video frame in the above undetected video frame sequence: The first sub-step involves inputting the aforementioned undetected video frames and the aforementioned historical frame sequence into a pre-trained image feature extraction model to obtain undetected video frame feature data corresponding to the undetected video frames and historical frame feature data sequences corresponding to the aforementioned historical frame sequence. The image feature extraction model can be a deep learning model that takes video frame images as input and outputs image feature vectors, transforming the video frames to be detected into understandable feature vectors representing the graphical content of the video frames to be detected. The deep learning model can include, but is not limited to, one of the following: a convolutional neural network model or a Vision Transformer (ViT) model. The undetected video frame feature data can be feature vectors containing content features of the video frames to be detected, obtained after inputting the undetected video frames into the image feature extraction model. These content features can be attributes such as edges, textures, and colors of the images in the video frames to be detected. Each historical frame feature data in the aforementioned historical feature data sequence can be a feature vector obtained after inputting the aforementioned historical frames into the image feature extraction model. The aforementioned historical frame feature data sequence can be an arrangement of historical frame feature data according to the order of the corresponding historical frames in the historical frame sequence. Each historical frame feature data in the above historical frame feature data sequence corresponds to a historical frame in the historical frame sequence.

[0045] The second sub-step involves generating a similarity score between each fault frame feature data point in the aforementioned fault frame feature data sequence and the aforementioned undetected video frame feature data, which is then used as the fault similarity score. This similarity score can be the cosine similarity between the fault frame feature data and the undetected video frame feature data.

[0046] The third sub-step involves determining the obtained fault similarity scores as a fault similarity set.

[0047] The fourth sub-step involves, in response to determining that at least one fault similarity value in the aforementioned fault similarity set is greater than a preset fault similarity value, deleting the undetected video frame from the aforementioned undetected video frame sequence to update the undetected video frame sequence. The preset fault similarity value can be a pre-set similarity value used as a threshold. In practice, when at least one fault similarity value in the aforementioned fault similarity set is greater than the preset fault similarity value, the executing entity can delete the undetected video frame from the undetected video frame sequence, thus removing the undetected video frame from the sequence and obtaining an updated undetected video frame sequence after deleting the undetected video frame.

[0048] The fifth sub-step, in response to determining that there is no fault similarity greater than a preset fault similarity in the aforementioned fault similarity set, generates a similarity between the historical frame feature data and the undetected video frame feature data for each historical frame feature data in the aforementioned historical frame feature data sequence. This historical similarity can be a cosine similarity value between the historical frame feature data and the undetected video frame feature data. In practice, when all fault similarity values ​​in the aforementioned fault similarity set are less than the preset fault similarity value, the executing entity can calculate a cosine similarity value between each historical frame feature data and the undetected video frame feature data in the aforementioned historical frame feature data sequence, and then use each obtained cosine similarity value as a historical similarity value.

[0049] The sixth sub-step involves determining the obtained historical similarities as a set of historical similarities.

[0050] The seventh sub-step involves, in response to determining that at least one historical similarity in the aforementioned historical similarity set is greater than a preset historical similarity, deleting the undetected video frame from the aforementioned undetected video frame sequence to update the undetected video frame sequence. The preset historical similarity can be a pre-set similarity value used as a threshold. In practice, when at least one historical similarity in the aforementioned historical similarity set has a similarity value greater than the preset historical similarity value, the executing entity can delete the undetected video frame from the aforementioned undetected video frame sequence, so that the undetected video frame no longer exists in the undetected video frame sequence, resulting in an updated undetected video frame sequence after deleting the undetected video frame.

[0051] The second step is to identify the updated sequence of undetected video frames as the sequence of video frames to be detected.

[0052] Step 105: For each video frame in the sequence of video frames to be detected, perform the following steps: Step 1051: Perform target object detection processing on the video frame to be detected to obtain a target object information set as the target object information set to be corrected.

[0053] In some embodiments, the execution entity may perform target object detection processing on the video frame to be detected to obtain a target object information set as a target object information set to be corrected. The target object information may be the target object detection information obtained after performing target object detection processing on the video frame to be detected.

[0054] In some optional implementations of certain embodiments, the execution entity may perform target object detection processing on the video frame to be detected through the following steps to obtain a target object information set as a target object information set to be corrected: The first step is to adjust the size of the video frame to be detected to a preset size. The preset size can represent the size of the transformed image, and can be the horizontal pixel value × the vertical pixel value of the image. For example, the preset size can be 1024 × 1024.

[0055] The second step is to perform pixel normalization on the resized video frame to be tested, resulting in a standardized video frame. In practice, the execution entity can perform the following pixel processing steps for each pixel in the resized video frame to be tested: First, the execution entity converts each integer value in the RGB values ​​of the pixel into a floating-point number. Then, for each converted RGB value, each floating-point RGB value is divided by 255 to obtain the standardized pixel value. Next, each pixel in the video frame to be tested is determined as a standardized video frame using the above-processed video frame. The pixel value of each pixel in the standardized video frame is within the range [0, 1]. As an example, the RGB value of a pixel in the above video frame to be detected is (R: 200, G: 100, B: 50), that is, the RGB value is (200, 100, 50), which is converted to a floating-point RGB value of (200.0, 100.0, 50.0). Dividing each value of the converted floating-point RGB value by 255, the normalized pixel value is (0.7843, 0.3922, 0.1961).

[0056] The third step involves inputting the standardized video frames into a pre-trained target object detection model to obtain a target object information set. Each target object information in this set can be the output of the target detection model, representing the information corresponding to a target object in the standardized video frame. Each target object information set can include an object identifier, target object candidate bounding box information, and target confidence. Each target object information set has a corresponding generation time. The target object detection model can be a neural network model that takes video image patches as input and outputs target detection information (e.g., R-CNN model, YOLO model). The target object candidate bounding box information can represent the position of the candidate bounding box that includes the target object. This information can include the coordinates of the top-left vertex, bottom-left vertex, top-right vertex, and bottom-right vertex of the target object candidate bounding box. The top-left vertex coordinates represent the coordinates of the top-left vertex of the target object candidate bounding box within the standardized video frame. The bottom-left vertex coordinates represent the coordinates of the bottom-left vertex of the target object candidate bounding box within the standardized video frame. The coordinates of the top-right vertex of the target object candidate box mentioned above represent the coordinates of the top-right vertex of the target object candidate box in the standardized video frame. The coordinates of the bottom-right vertex of the target object candidate box mentioned above represent the coordinates of the bottom-right vertex of the target object candidate box in the standardized video frame. The object identifier mentioned above can be information representing the category of the detected object (e.g., vehicle, pedestrian, traffic light) output by the object detection model. The object confidence score mentioned above can be information representing the reliability of the relevant content output by the object detection model regarding the detected object.

[0057] Fourth, perform the following steps for each target object information in the above target object information set: The first sub-step involves, in response to determining that the target confidence level included in the aforementioned target object information is greater than a preset reliable confidence level, classifying the aforementioned target object information as reliable target object information and deleting it from the aforementioned target object information set to be corrected, thereby updating the aforementioned target object information set. The preset reliable confidence level can be a pre-set confidence level value used for comparison. In practice, when the confidence level represented by the target confidence level included in the aforementioned target object information is greater than the reliable confidence level, the executing entity can classify the aforementioned target object information as target object information to be corrected. Then, the aforementioned target detection information is deleted from the aforementioned target object information set to be corrected, so that it no longer exists in the aforementioned target object information set to be corrected.

[0058] The second sub-step involves outputting and displaying the aforementioned reliable target object information. In practice, the executing entity can output the reliable target object information to a preset output terminal, and then display the object identifier and the area represented by the target object candidate box, which are included in the reliable target object information, in the video frame to be detected. The video frame to be detected, displaying the reliable target object information, is then displayed on the preset output terminal. The preset output terminal can be a monitor.

[0059] The third sub-step involves determining that the target confidence level included in the aforementioned target object information is greater than a reference confidence level and less than a preset reliable confidence level, thereby identifying the aforementioned target object information as target object information to be corrected. The reference confidence level can be a pre-set minimum confidence level value. The preset reliable confidence level can be a pre-set confidence level value used for comparison. It should be noted that when the target confidence level included in the aforementioned target object information is less than the reference confidence level, no further steps are performed on the aforementioned target object information.

[0060] The fifth step is to determine the information of each target object to be calibrated as a set of target object information to be calibrated.

[0061] Optionally, the above reference confidence level can be obtained through the following steps: The first step is to obtain the preset confidence level, confidence threshold range, and start detection time point. The preset confidence level can be a pre-set confidence value. The confidence threshold range can be a pre-set range within which the confidence level can vary, including a minimum and maximum confidence value. The start detection point can be the time point at which detection begins in the currently undetected video frame sequence. In practice, the execution entity can obtain the preset confidence level, confidence threshold range, and start detection time point from a storage device.

[0062] The second step is to generate a detection period based on the aforementioned start detection time and generation time. In practice, the executing entity can determine the time period between the start detection time and the generation time, and use this time period as the detection period. For example, if the start detection time is 11:07 and the generation time is 11:15, then the generated detection period can be 11:07-11:15.

[0063] The third step involves generating a sequence of periodic time segments based on the aforementioned detection time period and preset cycle length. The preset cycle length can be a pre-set numerical value representing the length of one periodic unit. The periodic time segment can be a time interval representing a period of time. The sequence of periodic time segments is formed by arranging the periodic time segments in the chronological order of the time intervals they represent. In practice, the executing entity can first divide the aforementioned detection time period according to the preset cycle length to obtain various periodic time segments. Specifically, if the last periodic time segment obtained after dividing the detection time period according to the preset cycle length is shorter than the preset cycle length, it is discarded. Then, the various periodic time segments are arranged in the chronological order of the time intervals they represent to obtain the sequence of periodic time segments. As an example, the detection time period mentioned above can be 11:07-11:16, and the preset cycle length can be two minutes. The resulting cycle time periods are: Cycle Time Period 1 {11:07-11:09}, Cycle Time Period 2 {11:09-11:11}, Cycle Time Period 3 {11:11-11:13}, and Cycle Time Period 4 {11:13-11:15}. The time period 11:15-11:16 does not meet the requirement of a two-minute cycle length and is therefore discarded. The resulting cycle time period sequence is then {Cycle Time Period 1, Cycle Time Period 2, Cycle Time Period 3, Cycle Time Period 4}.

[0064] The fourth step involves identifying at least one target object information generated within the aforementioned detection period as at least one reference object information. In practice, the executing entity can select at least one target object information whose generation time falls within the time period represented by the aforementioned detection period. Then, this at least one target information is identified as at least one reference object information. For example, the generation time of target object information 1 could be 10:58, the generation time of target object information 2 could be 11:05, the generation time of target object information 3 could be 11:08, and the generation time of target object information 4 could be 11:10, with an interval of 11:07-11:15. In this case, the identified at least one reference target would be target object information 3 and target object information 4.

[0065] Fifth, based on the at least one reference object information mentioned above, a reference object information sequence is generated. In practice, the executing entity arranges the at least one reference object information obtained according to the chronological order of its generation time to obtain the reference object information sequence.

[0066] Step 6: Based on the aforementioned reference object information sequence, generate at least one full confidence score. In practice, for each piece of reference object information in the aforementioned reference object information sequence, firstly, the executing entity can determine the target confidence score included in the aforementioned reference object information as the information confidence score. Next, the aforementioned information confidence score, the preset confidence score, and the timestamp corresponding to the reference object information are combined into an information group (e.g., (information confidence score, preset confidence score, timestamp)). Then, this information group is used as the full confidence score information. Finally, at least one full confidence score information is obtained. The timestamp can be the generation time of the target object information corresponding to the aforementioned reference object information. Each piece of full confidence score information includes the information confidence score, the preset confidence score, and the timestamp. Each piece of full confidence score information in the aforementioned full confidence score information sequence corresponds to one piece of reference object information. As an example, the above sequence of reference object information can be {Reference Object Information 1, Reference Object Information 2, Reference Object Information 3}. Reference Object Information 1 can have a target confidence level of 0.80 and a timestamp corresponding to Reference Object 1 of 10:55. Reference Object Information 2 can have a target confidence level of 0.87 and a timestamp corresponding to Reference Object 2 of 10:56. Reference Object Information 3 can have a target confidence level of 0.92 and a timestamp corresponding to Reference Object 3 of 10:58. If the above-mentioned preset confidence level can be 0.85, then the full confidence information 1 corresponding to reference object information 1 is (0.80, 0.85, 10:55), the full confidence information 2 corresponding to reference object information 2 is (0.87, 0.85, 10:56), and the full confidence information 3 corresponding to reference object information 3 is (0.92, 0.85, 10:58). At least one full confidence information is obtained as full confidence information 1, full confidence information 2, and full confidence information 3.

[0067] Step 7: Based on the above-mentioned at least one full confidence information, generate the average confidence score. In practice, the executing entity can first determine the sum of the confidence scores of each piece of information included in the above-mentioned at least one full confidence information as a first value. Then, the quotient obtained by dividing the above-mentioned first value by the number of full confidence information pieces in the full confidence information sequence is used as the average confidence score. As an example, the above-mentioned at least one full confidence information is confidence information 1, confidence information 2, and confidence information 3. The confidence score of the information included in confidence information 1 is 0.80, the confidence score of the information included in confidence information 2 is 0.87, and the confidence score of the information included in confidence information 3 is 0.92. The sum of the three confidence scores is 2.59. Then, 2.59 is divided by the number of confidence scores, which is 3, to obtain the average confidence score of 0.86.

[0068] Step 8: Based on the aforementioned periodic time segment sequence and the aforementioned reference object information sequence, generate at least one periodic object information. In practice, the executing entity first selects the last periodic time segment from the aforementioned periodic time segment sequence. Then, it filters at least one reference object information whose timestamp falls within the time interval represented by the last periodic time segment from the aforementioned reference object information sequence as a reference object information set. Then, for each reference object information in the reference object information set, the executing entity determines the target confidence level included in the reference object information as the periodic object confidence level. Next, the periodic object confidence level, along with the preset confidence level and the timestamp of the target object information corresponding to the reference object information, are combined to form an information group (e.g., (target confidence level, preset confidence level, timestamp)). Then, the aforementioned information group is used as the periodic object information. Finally, at least one periodic object information is obtained. Each periodic object information in the aforementioned at least one periodic object information includes a periodic object confidence level, a preset confidence level, and a timestamp. Each periodic object information in the aforementioned at least one periodic object information corresponds to one reference object information in the reference object information sequence. As an example, the preset confidence level can be 0.85, the above periodic time sequence can be {periodic time period 1, periodic time period 2, periodic time period 3, periodic time period 4}, the time interval represented by periodic time period 1 can be {11:07-11:09}, the time interval represented by periodic time period 2 can be {11:09-11:11}, the time interval represented by periodic time period 3 can be {11:11-11:13}, and the time interval represented by periodic time period 4 can be {11:13-11:15}, the above reference object information sequence can be {reference object information 1, reference object information 2, reference object information 3}, the timestamp corresponding to reference object information 1 is 11:08, and the included target confidence level is 0.88. The timestamp corresponding to reference object information 2 is 11:13 and the target confidence level is 0.85. The timestamp corresponding to reference object information 3 is 11:14 and the target confidence level is 0.83. The last periodic time period is periodic time period 4, which represents the time interval {11:13-11:15}. Then, at least one reference object information whose timestamp is within the time interval {11:13-11:15} is selected as reference object information 2 and reference object information 3. The at least one periodic object information is obtained as periodic object information 1 (0.85, 0.85, 11:13) corresponding to reference object information 2 and periodic object information 2 (0.83, 0.85, 11:15) corresponding to reference object information 3.

[0069] Step 9: Based on the at least one periodic object information mentioned above, generate a periodic object information sequence. In practice, the executing entity can sort the periodic object information in the at least one periodic object information in ascending order according to the numerical value of the confidence level of the periodic objects included in the periodic object information, and use the resulting sequence as the periodic object information sequence. As an example, the at least one periodic object information can be periodic object information 1, periodic object information 2, and periodic object information 3, where periodic object information 1 is (0.83, 0.85, 11:08), periodic object information 2 is (0.82, 0.85, 11:10), and periodic object information 3 is (0.90, 0.85, 11:12). Then the obtained periodic object information sequence is {periodic object information 2, periodic object information 1, periodic object information 3}.

[0070] Step 10: Based on the aforementioned periodic object information sequence, generate a filter water level sequence. Each filter water level in the sequence corresponds to a periodic object in the periodic object information sequence. In practice, for each periodic object in the periodic object information sequence, the executing entity first determines the index number of the periodic object in the periodic object sequence and the number of elements (i.e., periodic objects) in the sequence, using the ratio of the index number to the number as the filter water level. Then, the executing entity arranges the obtained filter water level sequences according to the order of the corresponding periodic object information in the periodic object information sequence to obtain the filter water level sequence.

[0071] Step 11: The periodic object information in the above periodic object information sequence whose confidence level is greater than the preset confidence level is determined as the standard object information. As an example, the preset confidence level can be 0.85, and the above periodic object information sequence can be {periodic object information 1, periodic object information 2, periodic object information 3}, where periodic object information 1 is (0.82, 0.85, 11:08), periodic object information 2 is (0.83, 0.85, 11:10), and periodic object information 3 is (0.90, 0.85, 11:12). Then, the determined standard object information is periodic object information 3, which is (0.90, 0.85, 11:12).

[0072] Step 12: Determine the filtration water level corresponding to the above standard object information as the actual filtration ratio.

[0073] Step 13: Iterate through the above-mentioned periodic object information sequence in sequence, and determine the periodic object information in which the confidence level of the first periodic object in the periodic object information sequence is greater than the preset confidence level as the standard object information.

[0074] Step fourteen: In response to determining that the actual filtering ratio does not meet the preset conditions, an updated confidence level is generated based on the average confidence level. The preset conditions can be that the actual filtering ratio is within a preset range. This preset range can be a pre-defined numerical range including a minimum and a maximum filtering ratio. In practice, when the actual filtering ratio is less than the minimum filtering ratio, the executing entity uses the product of the average confidence level and a first preset value as the updated confidence level. When the actual filtering ratio is greater than the maximum filtering ratio, the product of the average confidence level and a second preset value is used as the updated confidence level. The maximum value of the updated confidence level is the maximum confidence level, and the minimum value is the minimum confidence level. The first preset value can be a value greater than 1 and less than 2. The second preset value can be a value less than 1 and greater than 0.

[0075] Step 15: Determine the updated confidence level as the reference confidence level.

[0076] Step 1052: Based on the video frame to be detected, perform pre-correction processing on each target object information in the target object information set to be corrected to obtain pre-correction information.

[0077] In some embodiments, the execution entity may perform pre-correction processing on each piece of target object information in the set of target object information to be corrected based on the video frame to be detected, to obtain pre-correction information. Each piece of target object information to be corrected includes an object identifier, target object candidate bounding box information, and target confidence level.

[0078] In some optional implementations of certain embodiments, the execution entity may perform pre-correction processing on each piece of information about the target object to be corrected in the set of information about the target object to be corrected based on the video frame to be detected, thereby obtaining pre-correction information: The first step is to perform the following steps for each target object information in the above set of target object information to be corrected: The first sub-step is to determine the candidate bounding box information of the target object included in the target object information to be corrected as the filtering candidate bounding box information.

[0079] The second sub-step is to determine the image within the candidate box represented by the above-mentioned candidate box information in the video frame to be detected as the image block of the video frame to be detected.

[0080] The third sub-step involves determining the object identifier in the above-mentioned target object information as the object identifier to be judged.

[0081] The fourth sub-step involves performing grayscale discrimination processing on the aforementioned video frame image blocks to be detected and the aforementioned object identifiers to be discriminated against, based on a preset large model, to obtain object discrimination information. The preset large model can be a general multimodal large model, such as Qwen2.5-VL or DeepSeek-VL. In practice, firstly, the executing entity can determine the object identifiers to be discriminated against as... The image block of the video frame to be detected is determined as Then, by calling the grayscale discrimination instruction through a preset large model, the content in the above-mentioned detected video frame image block is judged to determine whether it is consistent with the object identifier, and the discrimination result is output. The above-mentioned grayscale discrimination instruction can be an instruction to perform binary discrimination on the grayscale features of the video frame image block to be detected. The above-mentioned discrimination result can be "TURE" or "FALSE", where "TURE" indicates that the content in the above-mentioned detected video frame image block is consistent with the object identifier, and "FALSE" indicates that the content in the above-mentioned detected video frame image block is inconsistent with the object identifier. Among them, the above-mentioned grayscale discrimination instruction can be:

[0082] in, To determine the result, Instructions for the function module that invokes the visual language model to perform inference and prediction. For example, "Does the content of the image block in the video frame to be detected match the identifier of the object to be identified?" The identifier of the object to be identified. For the video frame image block to be detected

[0083] The fifth sub-step, in response to determining that the above discrimination information meets the preset conditions, involves pre-correcting the candidate bounding box information in the target object information based on the video frame to be detected and the preset large model, to generate pre-correction information. This pre-correction information includes a pre-correction object identifier, pre-correction target object candidate bounding box information, and pre-correction target confidence. The preset conditions can be that the discrimination information is "TURE". In practice, when the executing entity determines that the discrimination information is "TURE", it first determines the video frame to be detected as... The candidate box information is determined as Then, by calling the candidate bounding box correction instruction through the preset large model, the candidate bounding box information in the above-mentioned target object information is corrected, and the corrected candidate bounding box information is used as pre-correction information. The above correction instruction can be expressed as follows:

[0084] in, This is the corrected candidate box information. Instructions for the function module that invokes the visual language model to perform inference and prediction. For prompt words (e.g., "Based on the provided original image I and original candidate box B, complete the candidate box correction for 'white sedan turning left'"), For candidate box information, The video frame to be detected.

[0085] Step 1053: Determine each pre-calibration information as a pre-calibration information set.

[0086] In some embodiments, the execution entity may determine the various pre-correction information as a set of pre-correction information.

[0087] In addressing the technical problems mentioned above, the following challenges arise when applying a solution to the specific scenario of real-time detection of video footage captured by cameras on highways: the types of targets to be detected on highways are relatively limited, the targets are clear and few in number, the scene is relatively simple, and a single calibration can achieve high accuracy. However, all pre-calibrated information in the pre-calibration information set undergoes discrimination optimization processing, resulting in the waste of computational resources and reduced detection efficiency for most high-accuracy information. This application scenario requires the following characteristics: high-confidence results in simple scenarios do not require discrimination optimization processing. Faced with these technical problems, we have decided to adopt the following solution: The first step involves feature extraction processing of the aforementioned video frames to be detected, yielding feature information including image entropy and edge density. In practice, the executing entity first processes the video frames using classic image processing algorithms (e.g., weighted average, simple average) to obtain a grayscale image. Then, the grayscale histogram of the grayscale image is obtained by calling the `cv2.calcHist(...)` command in OpenCV. Next, for each grayscale level in the histogram, the probability of that grayscale level is obtained by dividing the number of pixels at that grayscale level by the total number of pixels. Then, the probability of the grayscale set and the grayscale levels are substituted into the Shannon entropy formula to obtain the image entropy. Canny edge detection is then performed on the grayscale image to obtain a binarized edge image. Finally, the total number of pixels and the number of non-black pixels (i.e., the number of edge pixels; after Canny edge detection, edges are white and the background is black) are determined in the binarized edge image. Finally, the edge density is obtained by dividing the number of edge pixels by the total number of pixels in the image. The grayscale histogram above includes each gray level and the number of pixels corresponding to each gray level. The gray levels range from 0 to 255, corresponding to all brightness levels of the grayscale image. The Shannon entropy formula is expressed as:

[0088] in, Represents image entropy, Indicates grayscale level. Indicates the first The probability of a gray level (i.e., the proportion of pixels at that gray level to the total number of pixels in the image).

[0089] The second step involves generating a feature vector for the video frame to be detected based on the aforementioned feature information. In practice, the executing entity performs Z-score normalization on the image entropy and edge density to obtain normalized entropy and normalized edge density. These normalized entropy and normalized density values ​​are then arranged in a fixed order to form a one-dimensional vector as the feature vector of the video frame to be detected. This fixed order can be normalized entropy first, followed by normalized edge density. For example, if the normalized entropy is 1.0 and the normalized edge density is 0.3, the one-dimensional vector can be (1.0, 0.3).

[0090] The third step involves inputting the feature vector of the video frame to be detected into a pre-trained scene complexity score model to obtain a scene complexity score. This pre-trained scene complexity score model can be a regression model (e.g., random forest, simple fully connected network) that takes the feature vector of the video frame to be detected as input and outputs the scene complexity score. Optionally, the scene complexity score model can be trained through the following steps: First, obtain a sample set. The samples in the sample set include sample video frame feature vectors and sample scene scores corresponding to the sample scene images. Second, perform the following training steps based on the sample set: First sub-step: input the sample video frame feature vector of at least one sample in the sample set into the initial neural network to obtain the sample predicted scene score corresponding to each of the at least one sample. Second sub-step: compare the sample predicted scene score corresponding to each of the at least one sample with the corresponding sample scene score. In practice, the execution entity can calculate the difference between the sample predicted scene score and the sample scene score as the comparison result. Third sub-step: determine whether the initial neural network has reached the preset optimization objective based on the comparison result. The optimization objective mentioned above could be whether the difference between the predicted scene score and the actual scene score is within a preset error value. The fourth sub-step, in response to determining that the initial neural network has achieved the above optimization objective, uses the initial neural network as the trained scene complexity score model. The fifth sub-step, in response to the initial neural network not achieving the above optimization objective, adjusts the network parameters of the initial neural network, and uses unused samples to form a sample set, then executes the above training steps again. As an example, the back propagation algorithm (BP algorithm) and gradient descent methods (such as mini-batch gradient descent algorithm) can be used to adjust the network parameters of the initial neural network.

[0091] Fourth, in response to determining that the scenario complexity score is less than a preset first threshold, perform the following steps for each piece of pre-correction information in the pre-correction information set: The first sub-step is to determine the target confidence level included in the above pre-correction information as the screening target confidence level; The second sub-step is to determine, in response to the determination that the confidence level of the above-mentioned screening target is greater than the preset screening confidence level, to determine the above-mentioned target object information to be corrected as output target object information, and to delete the above-mentioned target object information to be corrected from the above-mentioned pre-correction information set, so as to update the above-mentioned pre-correction information set.

[0092] The fifth step is to output and display the above-mentioned outputtable target object information.

[0093] The above technical solution and its related content, combined with steps 1033-1037, serve as an inventive point of this disclosure, solving the technical problem of "leading to wasted computing resources and reduced detection efficiency." Factors leading to wasted computing resources and reduced detection efficiency often include: in cases where the scene is clear and simple, a single correction can achieve a high accuracy rate; all pre-correction information in the pre-correction information set undergoes discrimination optimization processing, resulting in most information with high accuracy still undergoing discrimination optimization processing, leading to wasted computing resources and reduced detection efficiency. Solving these factors can reduce wasted computing resources and improve detection efficiency. To achieve this effect, firstly, the above-mentioned video frame to be detected undergoes feature extraction processing to obtain the feature information of the video frame to be detected. Thus, the feature information of the video frame to be detected, representing the image characteristics of the video frame to be detected, can be obtained. Then, based on the above-mentioned feature information of the video frame to be detected, a feature vector of the video frame to be detected is generated. Thus, a feature vector of the video frame to be detected that can be input into the scene complexity score model is obtained. The above-mentioned feature vector of the video frame to be detected is input into the pre-trained scene complexity score model to obtain the scene complexity score. Thus, the scene complexity score representing the scene complexity in the video frame to be detected can be obtained. Next, in response to determining that the scene complexity score is less than a preset first threshold, the following steps are performed for each piece of pre-correction information in the pre-correction information set: First, the target confidence level included in the pre-correction information is determined as the screening target confidence level. In response to determining that the screening target confidence level is greater than a preset screening confidence level, the target object information to be corrected is determined as output target object information, and the target object information to be corrected is deleted from the pre-correction information set to update the pre-correction information set. Thus, an updated pre-correction information set for discrimination optimization processing can be obtained. Finally, the output target object information is output and displayed. Thus, output target object information with a certain degree of reliability can be obtained. Therefore, combined with the content of step 1054, the images in the video frame to be detected are detected. When the scene is relatively simple, the confidence level of the pre-correction information obtained after one pre-correction processing is judged. When the confidence level reaches a certain value, it proves that the result represented by the pre-correction information is relatively reliable and can meet the requirements. Subsequent discrimination optimization processing is no longer performed, reducing the waste of computing resources and improving detection efficiency.

[0094] Step 1054: Based on the video frame to be detected, perform discrimination and optimization processing on each pre-correction information in the pre-correction information set to obtain optimized target object detection information.

[0095] In some embodiments, the execution entity may perform discrimination and optimization processing on each pre-correction information in the pre-correction information set based on the video frame to be detected, so as to obtain optimized target object detection information.

[0096] In some optional implementations of certain embodiments, the execution entity may perform discriminative optimization processing on each pre-correction information in the pre-correction information set based on the video frame to be detected through the following steps to obtain optimized target object detection information: The first step is to determine the candidate bounding box information of each pre-correction target object included in each pre-correction information set as the candidate bounding box information to be judged.

[0097] The second step is to perform the following discrimination optimization process on each of the above candidate box information: The first sub-step involves generating response information based on the aforementioned video frame to be detected and the aforementioned candidate bounding box information, using a pre-set large model. In practice, the aforementioned execution entity first determines the video frame to be detected as... The candidate box information to be judged is determined as Then, the above can be... and the above The input is placed into a preset large model, and the response information is obtained through the following preset invocation command. The preset invocation command uses... express:

[0098] in, Reply information for multimodal large models, The probability value for each word element. These are instructions for the functional modules that invoke visual language models to perform inference and prediction. For example, "Based on the video frame to be detected and the candidate bounding box information to be judged, determine whether the candidate bounding boxes described by the candidate bounding box information accurately cover the target and provide more accurate candidate bounding box information" (e.g., "Based on the video frame to be detected and the candidate bounding box information to be judged, determine whether the candidate bounding boxes accurately cover the target and provide more accurate candidate bounding box information"). The video frame to be detected. Information for candidate boxes to be judged.

[0099] The second sub-step involves parsing the aforementioned response information to obtain the candidate box correction result information. This candidate box correction result information can be information representing the corrected candidate box region output by the large model, including the coordinates of the four vertices of the candidate box. In practice, the executing entity can input the aforementioned response information into a preset large model and obtain the discrimination information and candidate box correction result information through parsing instructions. The parsing instructions can be expressed as follows:

[0100]

[0101] in, This indicates a reply message. Indicates the discriminant information, This indicates the information regarding the candidate box correction results. This indicates an instruction to filter out the discriminant information from the response message. This indicates an instruction to filter out the correction information from the candidate boxes in the response message.

[0102] The third sub-step involves determining the object identifier and the detection box correction result information included in the pre-calibration information as the optimized target object detection information.

[0103] In addressing the aforementioned technical problems by employing technical solutions, the following challenges arise in the application scenario: real-time monitoring of traffic intersection video feeds under extreme weather conditions. Specifically, in severe weather, the camera feeds are often blurry, object edges are indistinct, and when using a pre-defined large model to optimize candidate bounding boxes, visual illusions can occur, leading to a significant deviation between the final output and the actual result, thus reducing detection accuracy. To address these issues and improve detection accuracy, we have decided to adopt the following solution to obtain optimized target object detection information: The first step is to set the historical response information to empty. This historical information can be previous responses from the multimodal large model. In practice, the executing entity can set the historical response information to empty.

[0104] The second step is to determine the candidate box information of each pre-correction target object included in each pre-correction information in the above pre-correction information sequence as the candidate box information to be judged.

[0105] The third step involves performing the following discrimination optimization process on each of the above candidate bounding box information using a pre-set large model: The first sub-step involves generating response information and a sequence of response information word probabilities based on historical response information, the aforementioned video frames to be detected, and the aforementioned candidate bounding boxes to be discriminated. The response information can be the response content output by a pre-defined large model. The word probabilities can be the confidence scores of each word in the response information. Each word probability corresponds to a word in the response information. The sequence of response information word probabilities can be composed of the probabilities of each word, arranged according to the order in which the corresponding word in the response information appears in the aforementioned response information. In practice, the execution entity first determines the video frames to be detected as... The candidate box information to be judged is Then, the aforementioned historical response information and the aforementioned video frames to be detected are... And the above candidate box information to be judged The following preset instructions are input into a preset large model to obtain the response information and the probability of each response word. Then, the probability of each response word is arranged according to the order in which one of its corresponding words appears in the aforementioned response information to obtain a sequence of response word probabilities. The preset instructions are as follows:

[0106] in, Reply information for multimodal large models, The probability value for each word element. These are instructions for the functional modules that invoke visual language models to perform inference and prediction. For example, "Based on the video frame to be detected, the candidate bounding box information to be judged, and the historical response information, determine whether the candidate bounding box information accurately covers the target and provide more accurate candidate bounding box information" (e.g., "Based on the video frame to be detected, the candidate bounding box information to be judged, and the historical response information, determine whether the candidate bounding box information accurately covers the target and provide more accurate candidate bounding box information"). The video frame to be detected. Information for candidate boxes to be judged. This is a historical reply message.

[0107] The second sub-step involves generating an average probability value for the response information based on the aforementioned sequence of response information token probabilities. In practice, the average value represented by the probabilities of each response information token generated by the aforementioned execution entity is used as the average probability value of the response information.

[0108] The third sub-step involves parsing the aforementioned response information to obtain discriminant information and candidate box correction result information. The candidate box correction result information can be information representing the corrected candidate box region output by the large model, including the coordinates of the four vertices of the candidate box. In practice, the executing entity inputs the aforementioned response information into a preset large model and obtains the discriminant information and candidate box correction result information through parsing instructions. The parsing instructions are expressed as follows:

[0109]

[0110] in, This indicates a reply message. Indicates the discriminant information, This indicates the information regarding the candidate box correction results. This indicates an instruction to filter out the discriminant information from the response message. This indicates an instruction to filter out the correction information from the candidate boxes in the response message.

[0111] The fourth sub-step involves generating a candidate box intersection-union ratio (IUU) based on the aforementioned candidate box correction results and the aforementioned candidate box information to be determined. In practice, the executing entity substitutes the candidate box regions represented by the aforementioned candidate box correction results and the candidate box regions represented by the aforementioned candidate box information to be determined into a preset IUU formula, and the resulting IUU value is used as the candidate box IUU value. The preset IUUU formula uses... Represented as: ,in, This represents the area of ​​the candidate box region as characterized by the candidate box correction result information. This represents the area of ​​the candidate box region represented by the candidate box information to be judged. This represents the intersection of the area of ​​the candidate box region represented by the candidate box correction result information and the area of ​​the candidate box region represented by the candidate box information to be judged.

[0112] The fifth sub-step involves determining that the intersection-union ratio of the candidate boxes is greater than a preset intersection-union ratio, and then identifying the response information as historical response information. The preset intersection-union ratio can be a pre-set value used for comparison. In practice, when the intersection-union ratio of the candidate boxes is less than the preset intersection-union ratio, the executing entity does not operate on the historical response information, and the historical response information remains empty.

[0113] The sixth sub-step involves, in response to determining that the average probability value of the aforementioned response information is less than a preset average probability value, identifying the aforementioned candidate box correction result information as candidate box information to be judged, updating the candidate box information to be judged, and performing the aforementioned discrimination optimization process again on the updated candidate box information to be judged. The preset average probability value can be a pre-set value used as a minimum standard.

[0114] The fourth step is to determine the object identifier and detection box correction result information included in the pre-correction information as the optimized target object detection information in response to the determination that the average probability value of the above response information is greater than or equal to the preset average probability value.

[0115] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "low detection accuracy". Factors leading to low detection accuracy often include: in poor weather, the image captured by the camera is blurry, and object edges are unclear; when the clarity of the video frame to be detected is low, the multimodal operation is prone to illusions when calling the preset large model for discrimination optimization processing of candidate boxes, leading to a final output result that may completely deviate from the actual result, resulting in low detection accuracy. Solving these factors can improve detection accuracy. To achieve this, firstly, the historical response information is set to empty. Then, the candidate box information of each pre-correction target object included in each pre-correction information sequence is determined as the candidate box information to be discriminated. Thus, the candidate box information to be discriminated for discrimination optimization processing can be obtained. Then, for each of the candidate bounding boxes to be discriminated in the aforementioned candidate bounding box information, the following discrimination optimization processing is performed using a preset large model: First, based on the aforementioned historical response information, the aforementioned video frames to be detected, and the aforementioned candidate bounding box information, response information and response information word probability sequences are generated. This yields response information representing the discrimination optimization result and response information word probability sequences representing the reliability of each word in the result. Then, based on the aforementioned response information word probability sequences, an average probability value for the response information is generated. This yields the average probability value for the response information representing its reliability. Next, the aforementioned response information is parsed to obtain candidate bounding box correction result information. This yields candidate bounding box correction result information representing the candidate bounding boxes after discrimination optimization processing. Then, based on the aforementioned candidate bounding box correction result information and the aforementioned candidate bounding box information to be discriminated, a candidate bounding box intersection-union ratio (CUI) is generated. This yields the candidate bounding box intersection-union ratio, representing the degree of overlap between the candidate bounding boxes represented by the candidate bounding box correction result information in the response information obtained through discrimination optimization processing and the candidate bounding boxes represented by the candidate bounding boxes to be discriminated. Next, in response to determining that the intersection-union ratio (IU) of the candidate boxes is greater than a preset IU value, the response information is identified as historical response information. Thus, when there is no significant shift between the candidate boxes represented by the candidate box correction results obtained after the discriminative optimization process and the candidate boxes represented by the candidate box information to be discriminated, historical response information that can be used as a reference for the next discriminative optimization process is obtained. Then, in response to determining that the average probability value of the response information is less than a preset average probability value, the candidate box correction results are updated and identified as candidate box information to be discriminated, so as to update the candidate box information to be discriminated and to perform the discriminative optimization process again on the updated candidate box information to be discriminated. Thus, when the average probability value of the response information in the current discriminative optimization process does not reach the preset average probability value, candidate box information to be discriminated can be obtained for performing discriminative optimization again.Finally, in response to the determination that the average probability value of the above response information is greater than or equal to the preset average probability value, the object identifier and detection box correction result information included in the above pre-correction information are determined as optimized target object detection information. When the average probability value of the response information obtained from this discrimination and optimization process reaches the preset average probability value, object identifier and detection box correction result information that can be output as optimized target object detection information are obtained. Also, because the average probability value of the response information obtained from the discrimination and optimization process does not reach the preset average probability value, the discrimination and optimization process is repeated. This avoids outputting unreliable response information from the large model when the image clarity is low. By repeatedly performing discrimination and optimization processing, response information that meets the preset average probability value is obtained, thus obtaining a more reliable discrimination and optimization processing result. Therefore, combined with the content of step 106, more reliable optimized target object detection information is obtained. Based on this, more reliable optimized target object detection information is output, improving the accuracy of the detection results.

[0116] Step 106: Output and display the obtained detection information for each optimized target object.

[0117] In practice, the aforementioned execution entity can output the detection information of each of the above-mentioned optimization target objects to a preset output terminal and display it, wherein the preset output terminal can be a display.

[0118] The above embodiments of this disclosure have the following beneficial effects: the target object detection method of some embodiments of this disclosure improves the accuracy of target object recognition and reduces the waste of computing resources. Specifically, the reason for the poor accuracy of object recognition and the waste of computing resources is that: pixel-level analysis is performed on the video represented by the directly acquired video stream data, and a single deep learning model is used to identify and frame all target objects of interest and their categories in the video. The training data and optimization targets of the single deep learning model are usually for specific scenes (such as daytime, sunny days, urban roads). When the scene changes significantly (such as sudden changes in lighting, severe weather, complex background), the accuracy of object recognition is poor. At the same time, before performing pixel-level analysis on the video stream data, multiple video frames with high similarity and no obvious difference in the video are not optimized and deduplicated. The similarity of multiple consecutive frames is very high. When performing pixel-level analysis, repeatedly processing these almost identical frames will waste computing resources. Based on this, the target object detection method of some embodiments of this disclosure first obtains the sequence of undetected video frames. Then, it obtains the preset fault frame feature data sequence and the detected video frame sequence. Therefore, a fault frame feature data sequence can be obtained for subsequent removal of problematic video frames from the undetected video frames, and a sequence of detected video frames can be obtained for filtering historical frames. Then, at least one detected video frame is filtered from the above-mentioned sequence of detected video frames to generate a historical frame sequence. This yields a historical frame sequence for removing undetected video frames from the undetected video frame sequence that are similar to historical frames. Then, based on the fault frame feature data sequence and the above-mentioned historical frame sequence, the undetected video frame sequence is optimized and deduplicated to obtain a processed undetected video frame sequence as the video frame sequence to be detected. This yields a video frame sequence to be detected that is free of faults and significantly different from historical frames, reducing resource waste caused by subsequent target object detection on multiple similar frames. Afterwards, for each video frame to be detected in the above-mentioned video frame sequence, the following steps are performed: First, target object detection processing is performed on the above-mentioned video frame to be detected to obtain a target object information set as the target object information set to be corrected. Therefore, after one target detection process, the video frame to be detected yields a set of target object information, including information about each target object in the video frame. This set serves as the target object information set to be corrected for subsequent correction processing. Next, based on the video frame to be detected, each target object information in the target object information set is pre-corrected to obtain pre-corrected information. This results in more accurate pre-corrected information after one correction. Then, each pre-corrected information is defined as a pre-corrected information set. Finally, based on the video frame to be detected, each pre-corrected information in the pre-corrected information set is subjected to discrimination and optimization processing to obtain optimized target object detection information.Therefore, optimized target object detection information after discriminative optimization processing can be obtained. Finally, the obtained optimized target object detection information is output and displayed. This yields a high-accuracy detection result after discriminative optimization processing. This is because by optimizing and deduplicating undetected video frames, multiple video frames with high similarity and no obvious differences or with malfunctions are deleted from the undetected video frames, resulting in a sequence of video frames to be detected. Then, the target object information set obtained from target object detection processing of each video frame in the sequence is pre-corrected, and the target object information in the pre-correction information set is corrected to obtain more accurate target object information as pre-correction information. Finally, combining the images in the video frames to be detected with the pre-correction information set, pre-correction processing and discriminative optimization processing are performed to obtain optimized target object detection information with improved detection accuracy.

[0119] Further reference Figure 2 As an implementation of the methods shown in the figures, this disclosure provides some embodiments of a plate information recognition device, which are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0120] like Figure 2 As shown, a target object detection device 200 in some embodiments includes: a first acquisition unit 201, a second acquisition unit 202, a generation unit 203, a first processing unit 204, a second processing unit 205, and an output unit 206. The first acquisition unit 201 is configured to acquire an undetected video frame sequence; the second acquisition unit 202 is configured to acquire a preset fault frame feature data sequence and a detected video frame sequence; the generation unit 203 is configured to filter at least one detected video frame from the detected video frame sequence to generate a historical frame sequence; the first processing unit 204 is configured to optimize and deduplicate the undetected video frame sequence based on the fault frame feature data sequence and the historical frame sequence to obtain a processed undetected video frame sequence as a video frame sequence to be detected; the second processing unit 205 is configured to process the undetected video frame sequence... For each video frame to be detected, the following steps are performed: target object detection processing is performed on the video frame to be detected to obtain a target object information set as a target object information set to be corrected; based on the video frame to be detected, pre-correction processing is performed on each target object information in the target object information set to be corrected to obtain pre-correction information; each pre-correction information is determined as a pre-correction information set; based on the video frame to be detected, discrimination optimization processing is performed on each pre-correction information in the pre-correction information set to obtain optimized target object detection information; the output unit 206 is configured to output and display the obtained optimized target object detection information.

[0121] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the method described above correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0122] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0123] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0124] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0125] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0126] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0127] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0128] Computer-readable media may be contained within an electronic device or may exist independently of the electronic device. A computer-readable medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: acquire an undetected video frame sequence; acquire a preset fault frame feature data sequence and a detected video frame sequence; select at least one detected video frame from the detected video frame sequence to generate a historical frame sequence; optimize and deduplicate the undetected video frame sequence based on the fault frame feature data sequence and the historical frame sequence to obtain a processed undetected video frame sequence as a video frame sequence to be detected; and for each video frame to be detected in the video frame sequence to be detected, perform the following steps: perform target object detection processing on the video frame to be detected to obtain a target object information set as a target object information set to be corrected; perform pre-correction processing on each target object information in the target object information set to be corrected based on the video frame to be detected to obtain pre-correction information; determine each pre-correction information as a pre-correction information set; perform discrimination optimization processing on each pre-correction information in the pre-correction information set based on the video frame to be detected to obtain optimized target object detection information; and output and display the obtained optimized target object detection information.

[0129] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first acquisition unit, a second acquisition unit, a generation unit, a first processing unit, a second processing unit, and an output unit. The names of these units do not necessarily limit the specific unit itself. For example, the generation unit may also be described as "a unit that optimizes and deduplicates the undetected video frame sequence based on the aforementioned fault frame feature data sequence and the aforementioned historical frame sequence to obtain a processed undetected video frame sequence as the video frame sequence to be detected."

[0132] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0133] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by arbitrary combinations of technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A target object detection method, comprising: obtaining an undetected video frame sequence; obtaining a preset fault frame feature data sequence and a detected video frame sequence; selecting at least one detected video frame from the detected video frame sequence to generate a history frame sequence; based on the fault frame feature data sequence and the history frame sequence, performing optimization and deduplication processing on the undetected video frame sequence to obtain a processed undetected video frame sequence as a to-be-detected video frame sequence; for each to-be-detected video frame in the to-be-detected video frame sequence, performing the following steps: performing target object detection processing on the to-be-detected video frame to obtain a target object information set as a to-be-corrected target object information set; based on the to-be-detected video frame, performing pre-correction processing on each to-be-corrected target object information in the to-be-corrected target object information set to obtain pre-correction information; determining each pre-correction information as a pre-correction information set; based on the to-be-detected video frame, performing discriminative optimization processing on each pre-correction information in the pre-correction information set to obtain optimized target object detection information; outputting and displaying each obtained optimized target object detection information.

2. The method of claim 1, wherein, The step of selecting at least one detected video frame from the detected video frame sequence to generate a history frame sequence comprises: determining a timestamp corresponding to a first undetected video frame in the undetected video frame sequence as a start timestamp; generating a history time interval based on a preset time length and the start timestamp; obtaining timestamps corresponding to each detected video frame in the detected video frame sequence as each detected video frame timestamp; generating a detected video frame timestamp sequence based on the each detected video frame timestamp; selecting at least one detected video frame timestamp with a timestamp within the history time interval from the detected video frame timestamp sequence; selecting at least one detected video frame corresponding to the at least one detected video frame timestamp from the detected video frame sequence as at least one history frame; adding the at least one history frame to a preset history frame sequence, wherein the existence time of each preset history frame in the preset history frame sequence in the preset history frame sequence is a preset time length; determining the preset history frame sequence after adding the at least one history frame as the history frame sequence.

3. The method of claim 1, wherein, The step of performing optimization and deduplication processing on the undetected video frame sequence based on the fault frame feature data sequence and the history frame sequence to obtain a processed undetected video frame sequence as a to-be-detected video frame sequence comprises: for each undetected video frame in the undetected video frame sequence, performing the following steps: inputting the undetected video frame and the history frame sequence into a pre-trained image feature extraction model to obtain undetected video frame feature data corresponding to the undetected video frame and a history frame feature data sequence corresponding to the history frame sequence; for each fault frame feature data in the fault frame feature data sequence, generating a similarity between the fault frame feature data and the undetected video frame feature data as a fault similarity; determine each of the obtained failure similarities as a failure similarity set; in response to determining that there is at least one failure similarity greater than the preset failure similarity in the failure similarity set, delete the undetected video frame from the undetected video frame sequence to update the undetected video frame sequence; in response to determining that there is no failure similarity greater than the preset failure similarity in the failure similarity set, generate, for each historical frame feature data in the historical frame feature data sequence, a similarity between the historical frame feature data and the undetected video frame feature data as a historical similarity; determine each of the obtained historical similarities as a historical similarity set; in response to determining that there is at least one historical similarity greater than the preset historical similarity in the historical similarity set, delete the undetected video frame from the undetected video frame sequence to update the undetected video frame sequence; determine the updated undetected video frame sequence as the to-be-detected video frame sequence.

4. The method of claim 1, wherein, The target object detection processing on the to-be-detected video frame obtains a target object information set as a to-be-corrected target object information set, which includes: adjusting the size of the to-be-detected video frame to a preset size; performing pixel normalization processing on the size-adjusted to-be-detected video frame to obtain a standardized video frame; inputting the standardized video frame into a pre-trained target object detection model to obtain a target object information set, wherein each target object information includes object identification, target object candidate box information, and target confidence; performing the following steps on each target object information in the target object information set: in response to determining that the target confidence included in the target object information is greater than a preset reliable confidence, determining the target object information as reliable target object information, and deleting the target object information from the to-be-corrected target object information set to update the target object information set; outputting and displaying the reliable target object information; in response to determining that the target confidence included in the target object information is greater than a reference confidence and less than a preset reliable confidence, determining the target object information as to-be-corrected target object information; determine each of the obtained to-be-corrected target object information as a to-be-corrected target object information set.

5. The method of claim 4, wherein, The target object information has a corresponding generation time, and the reference confidence is determined by the following steps: obtain a preset confidence, a confidence threshold range, and a start detection time point; generate a detection time period based on the start detection time point and the generation time; generate a periodic time period sequence based on the detection time period and a preset periodic length; determine at least one target object information generated within the detection time period as at least one reference object information; generate a reference object information sequence based on the at least one reference object information; generate at least one confidence fullness information based on the reference object information sequence, wherein each confidence fullness information includes information confidence, preset confidence, and timestamp; generate an average confidence based on the at least one confidence fullness information; generating at least one periodic object information based on the sequence of periodic time periods and the sequence of reference object information, wherein each periodic object information comprises a periodic object confidence, a preset confidence, and a timestamp; generating a sequence of periodic object information based on the at least one periodic object information; generating a sequence of filtered water level lines based on the sequence of periodic object information, wherein each filtered water level line in the sequence of filtered water level lines corresponds to one periodic object information in the sequence of periodic object information; sequentially traversing the sequence of periodic object information, and determining a periodic object information in which a first periodic object confidence greater than a preset confidence in the sequence of periodic object information as a standard object information; determining a filtered water level line corresponding to the standard object information as an actual filtering ratio; in response to determining that the actual filtering ratio does not satisfy a preset condition, generating an updated confidence based on the average confidence; determining the updated confidence as a reference confidence.

6. The method of claim 1, wherein, Each of the to-be-corrected target object information comprises object identification, target object bounding box information, and target confidence, and the pre-correction processing of each of the to-be-corrected target object information in the set of to-be-corrected target object information based on the to-be-detected video frame comprises: performing the following steps on each of the to-be-corrected target object information in the set of to-be-corrected target object information: determining the target object bounding box information included in the to-be-corrected target object information as screening bounding box information; determining an image in a bounding box represented by the screening bounding box information in the to-be-detected video frame as a to-be-detected video frame image block; determining the object identification in the to-be-corrected target object information as a to-be-judged object identification; performing grayscale judgment processing on the to-be-detected video frame image block and the to-be-judged object identification based on a preset large model to obtain object judgment information; in response to determining that the object judgment information satisfies a preset condition, performing pre-correction processing on the bounding box information in the to-be-corrected target object information based on the to-be-detected video frame and the preset large model to generate pre-correction information, wherein the pre-correction information comprises pre-correction object identification, pre-correction target object bounding box information, and pre-correction target confidence.

7. The method of claim 1, wherein, Each of the pre-correction information comprises object identification, target object bounding box information, and target confidence, and the judgment optimization processing of each of the pre-correction information in the set of pre-correction information based on the to-be-detected video frame comprises: determining each pre-correction target object bounding box information included in each of the pre-correction information in the set of pre-correction information as each to-be-judged bounding box information; performing the following judgment optimization processing on each of the to-be-judged bounding box information: generating reply information based on the to-be-detected video frame, the to-be-judged bounding box information, and a preset large model; performing analysis processing on the reply information to obtain bounding box correction result information; determining the object identification included in the pre-correction information and the detection frame correction result information as optimized target object detection information.

8. An object detection apparatus, comprising: a first obtaining unit configured to obtain an undetected video frame sequence; a second obtaining unit configured to obtain a preset failure frame feature data sequence and a detected video frame sequence; a generating unit configured to screen at least one detected video frame from the detected video frame sequence to generate a history frame sequence; a first processing unit configured to perform optimization and deduplication processing on the undetected video frame sequence based on the failure frame feature data sequence and the history frame sequence, to obtain a processed undetected video frame sequence as a to-be-detected video frame sequence; a second processing unit configured to perform the following steps on each to-be-detected video frame in the to-be-detected video frame sequence: performing object detection processing on the to-be-detected video frame to obtain a target object information set as a to-be-corrected target object information set; performing pre-correction processing on each to-be-corrected target object information in the to-be-corrected target object information set based on the to-be-detected video frame to obtain pre-correction information; determining each pre-correction information as a pre-correction information set; performing discriminative optimization processing on each pre-correction information in the pre-correction information set based on the to-be-detected video frame to obtain optimized object detection information; an output unit configured to output and display each obtained optimized object detection information.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

10. A computer readable medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method of any one of claims 1-7.

Images