A method and system for converting scene classification to scene temporal detection

Abstract

The application discloses a method and system for converting scene classification into scene time domain detection, and the method comprises the following steps: acquiring video data from a real-time video stream or a stored video file warehouse; performing frame extraction processing on the video data according to a preset frame rate to obtain a series of static images arranged in time sequence, and recording timestamp information of each image in the video; performing scene category identification on each image obtained through frame extraction to obtain a scene category label and a confidence value corresponding to each image; generating a preliminary scene time domain detection result according to the scene category label and the confidence value and in combination with a preset maximum interval image number of discontinuous scene recognition results; and performing multi-level filtering optimization on the preliminary scene time domain detection result to obtain a final scene time domain detection result. The application can improve the accuracy of scene time domain detection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, specifically a method and system for converting scene classification into scene temporal detection. Background Technology

[0002] The current method of converting indoor scene classification in kindergartens into scene temporal detection generally includes the following steps: (1) Define scene categories (such as indoor scenes in kindergartens: class, eating, group activities, sleeping), and train scene image classification models; (2) Obtain videos, which can be obtained from real-time video streams or video file warehouses; (3) Extract frames from the video at a certain frame rate to obtain a series of static images sorted by time, and record the time of each image in the video; (4) Perform scene recognition on the images; (5) Connect the times of images with consistent scene recognition results in time sequence into a time period as the scene temporal detection result.

[0003] Converting existing scene classification to scene temporal detection presents the following problems: errors in the recognition results of individual image scenes can cause a scene in temporal detection to be interrupted into multiple scenes or identified as an erroneous scene with a very short duration, resulting in low accuracy of scene temporal detection results. Summary of the Invention

[0004] This invention provides a method and system for converting scene classification into scene temporal detection based on a strategy. This method can solve the problem in the prior art where scene temporal detection results are interrupted or incorrect scenes are generated due to individual image recognition errors, thereby improving the accuracy and reliability of scene temporal detection.

[0005] Firstly, a strategy-based method for converting scene classification into scene temporal detection is provided, comprising the following steps: The method first acquires video data, which can be extracted from real-time video streams or from a stored video file repository; then, the video data is processed by frame extraction at a preset frame rate to obtain a series of static images arranged in chronological order, and the timestamp information of each image in the video is recorded; subsequently, scene category recognition is performed on each image obtained from the frame extraction to obtain the scene category label and its confidence value corresponding to each image; then, based on the scene category label and confidence value, combined with a preset maximum interval number of discontinuous scene recognition results, a preliminary scene temporal detection result is generated; finally, the preliminary scene temporal detection result is optimized through multi-level filtering to obtain the final scene temporal detection result.

[0006] In the solution provided by this invention, by setting a maximum interval of images for discontinuous scene recognition results, the interruption of scene temporal detection results due to individual image recognition errors can be effectively avoided. Simultaneously, by combining the minimum and maximum duration limits for scene temporal detection, short-term or excessively long detection results that do not conform to the logic of actual scenes are further eliminated. Furthermore, by introducing mutually exclusive scene filtering rules and reasonable constraints on scene time periods, the accuracy and practicality of scene temporal detection results can be further improved.

[0007] Secondly, a policy-based system for converting scene classification into scene temporal detection is provided, comprising: a video acquisition module for acquiring video data from real-time video streams or stored video file repositories; a frame extraction processing module for extracting frames from the video data at a preset frame rate and recording the timestamp information of each image; a scene recognition module for identifying the scene category of each image obtained from the frame extraction, and obtaining the scene category label and its confidence value corresponding to each image; a temporal detection module for generating preliminary scene temporal detection results based on the scene category label and confidence value, combined with a preset maximum interval number of discontinuous scene recognition results; and a result optimization module for performing multi-level filtering optimization on the preliminary scene temporal detection results to obtain the final scene temporal detection results.

[0008] Through the above technical solutions, the present invention provides a method and system for converting scene classification into scene temporal detection based on a strategy. This method can effectively solve the problem in the prior art where scene temporal detection results are interrupted or incorrect scenes are generated due to individual image recognition errors, thereby improving the accuracy and reliability of scene temporal detection. Attached Figure Description

[0009] Figure 1 A schematic diagram of the overall architecture of the policy-based scene classification and conversion to scene temporal detection system provided in this embodiment of the invention;

[0010] Figure 2 This is a schematic diagram of the workflow of the frame extraction processing module in an embodiment of the present invention, illustrating the process of extracting video data frames and recording timestamps;

[0011] Figure 3 This is a schematic diagram illustrating the working principle of the scene recognition module in this embodiment of the invention, including the process of generating scene category labels and confidence values;

[0012] Figure 4 This is a schematic diagram of the logical structure of the time-domain detection module in an embodiment of the present invention, illustrating the process of generating preliminary scene time-domain detection results;

[0013] Figure 5This is a schematic diagram of the operation flow of the scene continuity judgment unit in an embodiment of the present invention, describing the impact of the maximum number of interval images on scene continuity judgment;

[0014] Figure 6 This is a schematic diagram of the working mechanism of the mutual exclusion scene filtering unit in this embodiment of the invention, showing how to filter mutual exclusion scenes by confidence value;

[0015] Figure 7 This is a functional diagram of the time period rationality filtering unit in an embodiment of the present invention, illustrating how to remove scene segments that do not conform to the actual time period constraints;

[0016] Figure 8 A schematic diagram illustrating the overall process of the policy-based scene classification to scene temporal detection method provided in this embodiment of the invention;

[0017] Figure 9 A schematic diagram illustrating a specific process of the policy-based scene classification conversion to scene temporal detection method provided in an embodiment of the present invention;

[0018] Figure 10 Functional block diagram of a computer system provided in an embodiment of the present invention; Detailed Implementation

[0019] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] Example 1

[0021] This invention provides a method and system for converting scene classification into scene temporal detection based on a strategy. It addresses the problem of interrupted scene temporal detection results or the generation of incorrect scenes due to individual image recognition errors through a multi-layered filtering and optimization mechanism. The following is combined with... Figures 1 to 8 The specific embodiments of the present invention will be described in detail with reference to the markings in the accompanying drawings.

[0022] In practical applications, the system of this embodiment of the invention consists of multiple modules working collaboratively to complete the scene temporal detection task. First, the video acquisition module acquires video data from a real-time video stream or a stored video file repository. The video acquisition module is connected to the frame extraction processing module, transmitting the acquired video data to the frame extraction processing module for subsequent processing. For example... Figure 2As shown, the frame extraction module performs frame extraction on the video data according to a preset frame rate and records the timestamp information of each image. The frame extraction module contains a timer unit to accurately record the time position of each frame in the video and stores the timestamp information in a one-to-one correspondence with the image. The output of the frame extraction module is directly transmitted to the scene recognition module for further analysis.

[0023] The scene recognition module receives image data from the frame extraction processing module and performs scene category identification on it. For example... Figure 3 As shown, the scene recognition module contains a deep learning model. This model performs feature extraction and classification on the input images, generating scene category labels and their confidence values ​​for each image. The scene recognition module also includes a confidence filtering unit, which performs preliminary filtering of the recognition results. When the scene recognition confidence value of an image is higher than a preset threshold, the image is marked as a valid frame; otherwise, it is marked as an invalid frame. The output of the confidence filtering unit, along with the original recognition results, is passed to the temporal detection module.

[0024] The main function of the temporal detection module is to generate preliminary temporal detection results based on scene category labels and confidence values. For example... Figure 4 As shown, the internal structure of the time-domain detection module

[0025] This includes a scene continuity determination unit and a scene start time stamping unit. The scene continuity determination unit defines the maximum interval number of discontinuous scene recognition results (max_age) and initializes the interval number of discontinuous scene recognition results (age) to 0. When the current image does not belong to the current scene category, the scene continuity determination unit increments the age count by 1; when the current image belongs to the current scene category, age is reset to 0. If age exceeds max_age, the current scene is determined to have ended, and the timestamp corresponding to the last image belonging to that scene is recorded as the scene end time. The scene start time stamping unit is used to mark the timestamp corresponding to each scene category as the start time of that scene when it detects that the current image belongs to any scene category and there are no unfinished temporal detection results for that scene.

[0026] The initial scene temporal detection results are then passed to the result optimization module for multi-level filtering optimization. The result optimization module includes minimum and maximum duration filtering units, mutually exclusive scene filtering units, and time period rationality filtering units. The minimum and maximum duration filtering unit defines the minimum and maximum duration thresholds for each scene category and removes scene temporal segments from the initial scene temporal detection results that are less than the minimum duration threshold or greater than the maximum duration threshold. The working principle of the mutually exclusive scene filtering unit is as follows... Figure 6As shown, its processing logic is that when multiple mutually exclusive scenarios are detected within the same time period, only the scenario category with a confidence value higher than a preset confidence threshold is retained as the final detection result. For example, the "class" and "sleep" scenarios in a kindergarten indoor scene usually do not occur simultaneously. Therefore, when these two scenarios are detected to occur within the same time period, the mutually exclusive scenario filtering unit will prioritize the scenario category with the higher confidence value. The function of the time period reasonableness filtering unit is as follows: Figure 7 As shown, its function is to eliminate scene segments that do not conform to the constraints of the actual time period. For example, the "sleeping" scene in the indoor scene of a kindergarten usually occurs at noon. If a "sleeping" scene is detected between 10:00 and 10:30 am, the time period rationality filtering unit will regard it as an abnormal result and remove it.

[0027] like Figure 1 As shown, the various modules and units are connected via a data bus to form a complete system architecture. The video acquisition module transmits video data to the frame extraction processing module, which then transmits the extracted image data to the scene recognition module. The scene recognition module transmits the recognition results to the temporal detection module, which generates preliminary scene temporal detection results and transmits them to the result optimization module. Finally, the result optimization module outputs the scene temporal detection results in the form of a triple (scene_label, start_time, end_time).

[0028] In practical applications, this invention can be applied to a monitoring system for indoor scenes in kindergartens. For example, the system acquires video data from a real-time monitoring camera via a video acquisition module, and a frame extraction module extracts frames from the video at a rate of 10 frames per second, recording the timestamp information of each frame. The scene recognition module identifies the scene category of the extracted images, generating a scene category label and its confidence value for each image. Assume that the image recognition results within a certain time period show that the "class" scene appears continuously for 50 frames, the "sleeping" scene appears continuously for 10 frames, and the confidence value of the "class" scene is higher than that of the "sleeping" scene. The temporal detection module, based on the logic of the scene continuity judgment unit, sets max_age to 15 frames and, combined with the threshold settings of the minimum and maximum duration filtering units, generates preliminary temporal detection results. The result optimization module optimizes the preliminary results using a mutually exclusive scene filtering unit and a time period rationality filtering unit, ultimately outputting a scene temporal detection result in the form of a triplet. For example, the output might be (Class, 09:00:00, 09:10:00), indicating that the "Class" scenario lasts from 09:00:00 to 09:10:00.

[0029] The embodiments of the present invention improve the accuracy and reliability of scene temporal detection through a multi-level filtering optimization mechanism.

[0030] The following section provides further explanation of the specific implementation principle of this invention using a particular application scenario.

[0031] In the monitoring system for indoor scenes in kindergartens, the operation flow of this embodiment of the invention can be broken down into several steps for detailed explanation. First, the video acquisition module collects video data through a real-time monitoring camera and transmits the video stream to the frame extraction processing module. The frame extraction processing module performs frame extraction processing on the video data according to a preset frame rate (e.g., 10 frames per second), while simultaneously recording the timestamp information of each image. For example... Figure 2 As shown, the timer unit inside the frame extraction processing module accurately records the temporal position of each frame in the video and stores the timestamps in a one-to-one correspondence with the images. This process ensures the temporal correlation of subsequent scene recognition and temporal detection.

[0032] Subsequently, the frame extraction module transmits the extracted image data to the scene recognition module. The scene recognition module uses its internal deep learning model to perform feature extraction and classification operations on each image, generating corresponding scene category labels and their confidence scores. For example... Figure 3 As shown, the confidence filtering unit performs preliminary screening on the generated recognition results, retaining only valid frames with confidence values ​​higher than a preset threshold. This mechanism effectively reduces the interference of low-confidence frames on subsequent detection results, thereby improving the reliability of scene recognition.

[0033] Next, the temporal detection module generates preliminary temporal detection results based on the scene category label and confidence value. For example... Figure 4 As shown, the scene continuity judgment unit defines the maximum interval image count (max_age) for discontinuous scene recognition results and initializes the interval image count (age) to 0. When the current image does not belong to the current scene category, the scene continuity judgment unit increments the age count by 1; when the current image belongs to the current scene category, age is reset to 0. If age exceeds max_age, the current scene is determined to have ended, and the timestamp corresponding to the last image belonging to the scene is recorded as the scene end time. This logical design avoids the problem of interruption in the temporal domain detection results due to individual image recognition errors, thus ensuring the continuity of the detection results.

[0034] Meanwhile, the minimum and maximum duration filtering unit further optimizes the initial scene temporal detection results. This unit defines a minimum duration threshold and a maximum duration threshold for each scene category and removes scene temporal segments that do not meet these thresholds. For example, if the duration of a scene segment is less than the minimum duration threshold or greater than the maximum duration threshold, it is considered an abnormal result and removed. This mechanism effectively removes short or excessively long scene segments caused by misidentification, thereby improving the accuracy of the detection results.

[0035] After the initial temporal detection results are generated, the result optimization module performs multi-level filtering optimization on them. The working principle of the mutually exclusive scene filtering unit is as follows: Figure 6 As shown, when multiple mutually exclusive scenes are detected within the same time period, only the scene category with the higher confidence value is retained as the final detection result. For example, in a kindergarten indoor scene, the "class" and "sleep" scenes usually do not occur simultaneously. Therefore, when these two scenes occur within the same time period, the mutually exclusive scene filtering unit prioritizes the scene category with the higher confidence value. This design solves the problem of multiple scene overlap caused by scene category confusion, further improving the reasonableness of the detection results.

[0036] In addition, the functions of the time period rationality filtering unit are as follows: Figure 7 As shown, its function is to eliminate scene segments that do not conform to the actual time period constraints. For example, the "sleeping" scene in a kindergarten indoor scene usually occurs at noon. If a "sleeping" scene is detected between 10:00 and 10:30 am, the time period reasonableness filtering unit will regard it as an abnormal result and remove it. This mechanism effectively eliminates unreasonable scene time domain segments caused by misidentification, thereby improving the reliability of the detection results.

[0037] Finally, the scene temporal detection results, after multi-level filtering and optimization, are output as a triple (scene_label, start_time, end_time) by the result optimization module. All modules and units within the system are connected via a data bus, forming a complete automated processing flow. From video data acquisition to final result output, the system achieves efficient and accurate scene temporal detection.

[0038] As can be seen from the above steps, the embodiments of the present invention improve the accuracy and reliability of scene temporal detection through a multi-level filtering optimization mechanism. Especially in applications involving indoor scenes in kindergartens, the system can effectively address the problem of interrupted scene temporal detection results or the generation of incorrect scenes due to individual image recognition errors, thereby meeting practical application needs.

[0039] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0040] Example 2

[0041] like Figure 8 As shown, a method for converting policy-based scene classification into scene temporal detection is provided, including the following steps:

[0042] S110: The method first acquires video data, which can be extracted from a real-time video stream or from a stored video file repository;

[0043] S120: Next, the video data is processed by frame extraction according to a preset frame rate to obtain a series of static images arranged in chronological order, and the timestamp information of each image in the video is recorded.

[0044] S130: Subsequently, scene category recognition is performed on each image obtained by frame extraction to obtain the scene category label and its confidence value corresponding to each image;

[0045] S140: Then, based on the scene category label and confidence value, combined with the preset maximum interval number of discontinuous scene recognition results, a preliminary scene temporal detection result is generated;

[0046] S150: Finally, by performing multi-level filtering and optimization on the preliminary scene temporal detection results, the final scene temporal detection results are obtained.

[0047] In this embodiment of the invention, by setting a maximum interval of images for discontinuous scene recognition results, the interruption of scene temporal detection results due to individual image recognition errors can be effectively avoided. Simultaneously, by combining the minimum and maximum duration limits for scene temporal detection, short-term or excessively long detection results that do not conform to the logic of actual scenes are further eliminated. Furthermore, by introducing mutually exclusive scene filtering rules and reasonable constraints on scene time periods, the accuracy and practicality of scene temporal detection results can be further improved.

[0048] In some embodiments, step S140 specifically includes:

[0049] In the process of generating preliminary scene temporal detection results, the maximum interval number of discontinuous scene recognition results (max_age) is defined, and the interval number of discontinuous scene recognition results (age) is initialized to 0.

[0050] If the current image does not belong to the current scene category, increment the age count by 1;

[0051] When the current image belongs to the current scene category, reset age to 0;

[0052] When age exceeds max_age, the current scene is determined to end, and the timestamp of the last image belonging to the scene is recorded as the scene end time.

[0053] Step S140 establishes a buffer mechanism for scene temporal continuity by defining the maximum interval number of discontinuous scene recognition results (max_age) and initializing the interval count (age). When the current image does not belong to the current scene category, age is incremented progressively; when it does, the count is reset. Scene termination is only determined when the number of consecutive non-target scene images exceeds max_age. This mechanism effectively filters out scene interruptions caused by recognition errors in a single frame or a few frames (e.g., brief interference or noise), allowing the scene temporal detection results to tolerate a certain degree of recognition discontinuity. This enhances the robustness of the detection results to accidental recognition errors, ensuring that actual continuous scenes are not incorrectly segmented due to misjudgment of individual images. It improves the accuracy and stability of scene temporal boundary delineation, better reflecting the continuous characteristics of real scenes in the temporal dimension.

[0054] In some embodiments, step S140 further includes the following steps:

[0055] During the process of generating preliminary scene temporal detection results, for each scene category, if the scene recognition confidence value of the current image is higher than the preset threshold, the current image is marked as a valid frame of the scene category; otherwise, the current image is marked as an invalid frame, and subsequent judgment is made in combination with the maximum number of discontinuous images in the scene recognition results.

[0056] In some embodiments, step S140 further includes the following steps:

[0057] During the process of generating preliminary scene temporal detection results, for each scene category, when it is detected that the current image belongs to any scene category and there are no unfinished scene temporal detection results, the timestamp corresponding to the current image is marked as the start time of the scene.

[0058] This technical solution ensures the uniqueness and accuracy of the starting point of a scene's temporal segment by adhering to the rule of "marking the start time only when the current image belongs to a certain scene category and there are no unfinished temporal detection results for that scene." This avoids the redundancy of repeatedly marking the starting point of the same scene before it ends, and ensures that the initiation of each new scene segment is based on the actual scene transition boundary (rather than accidental recognition fluctuations in intermediate frames). This mechanism gives the generated preliminary scene temporal detection results a clear segmentation logic in the temporal dimension, conforming to the natural characteristics of continuous and non-overlapping scenes in reality, and providing a reliable time reference for subsequent steps such as duration calculation and mutual exclusion filtering, thereby improving the structural and logical consistency of the overall detection results.

[0059] In some embodiments, step S150 specifically includes:

[0060] After generating preliminary scene temporal detection results, define the minimum duration threshold and the maximum duration threshold for each scene category;

[0061] Scene time-domain segments that are less than the minimum duration threshold or greater than the maximum duration threshold in the preliminary scene time-domain detection results are removed.

[0062] This technical solution effectively filters out short-term invalid scenes caused by accidental recognition errors (such as instantaneous scenes caused by single-frame false detections) and excessively long abnormal scenes that violate objective laws (such as unreasonable scene continuations lasting for several hours) by defining minimum and maximum duration thresholds for scene categories and eliminating non-compliant time-domain segments. This makes the detection results more consistent with the logical characteristics of real scenes in the time dimension. This operation not only reduces false detections and redundant information, improving the rationality and credibility of scene time-domain detection results, but also lays a reliable foundation for subsequent steps such as mutually exclusive scene filtering and time-domain rationality analysis. Ultimately, it makes the output scene time-domain segments more in line with actual application needs (such as surveillance video analysis and intelligent video retrieval).

[0063] In some embodiments, step S150 further includes the following step:

[0064] After generating preliminary scene temporal detection results, the preliminary scene temporal detection results are optimized using mutually exclusive scene filtering rules;

[0065] When multiple mutually exclusive scenarios are detected within the same time period, only the scenario categories with confidence values ​​higher than the preset confidence threshold are retained as the final detection results.

[0066] This technical solution effectively resolves the logical contradiction of conflicting scenarios coexisting within the same time period by introducing mutually exclusive scene filtering rules and combining them with confidence threshold screening. Specifically, when mutually exclusive scenarios (such as "class" and "sleeping," or "meeting" and "free time") are detected to exist simultaneously, by eliminating scene categories with confidence scores below the threshold and retaining only results with higher confidence scores, the final detection results can be ensured to conform to the logical consistency of real-world scenarios (e.g., only one primary activity scenario can exist in a classroom at the same time). This operation not only eliminates semantic conflicts in the detection results and improves the accuracy and reliability of scene labels, but also provides a cleaner data source for subsequent applications such as video content analysis and event correlation judgment, thereby enhancing the usability of the entire scene temporal detection system in actual business and reducing the risk of misjudgment caused by logical contradictions.

[0067] In some embodiments, step S150 further includes the following step:

[0068] After generating preliminary scene temporal detection results, the preliminary scene temporal detection results are further optimized using the reasonableness constraints of scene time periods;

[0069] For each detected scene time-domain segment, if its time range does not match the reasonable time period that the scene category should occur in actual application, then the scene time-domain segment is removed from the preliminary detection results.

[0070] This technical solution optimizes the initial scene time-domain detection results by utilizing the reasonableness constraints of scene time periods. It matches the time range of the detected scene time-domain segments with the reasonable time period that the scene should occur in actual applications, and eliminates segments with inconsistent time ranges. This ensures that the detection results conform to real-world logic and objective laws in the time dimension (e.g., avoiding lunch break scenarios occurring in the morning or commuting scenarios occurring at night). It effectively improves the reasonableness, credibility, and practicality of scene time-domain detection results, providing more data support that meets actual needs for subsequent applications such as intelligent scheduling, behavior analysis, and anomaly warning. At the same time, it reduces the risk of misjudgment caused by contradictions in time logic and enhances the interpretability and business adaptability of the detection results.

[0071] In some embodiments, step S150 further includes the following step:

[0072] The final scene temporal detection result is output as a triple (scene_label, start_time, end_time), where scene_label represents the scene category label, start_time represents the scene start time, and end_time represents the scene end time.

[0073] In this embodiment of the invention, by outputting the scene temporal detection results in the form of triples, the category, start time and end time of each scene temporal segment can be intuitively displayed, which facilitates subsequent scene analysis and application.

[0074] Example 3

[0075] like Figure 9 As shown, this invention addresses the aforementioned problems by providing a method for converting policy-based scene classification into temporal scene detection. To solve the above technical problems, the technical solution provided by this invention is a method for converting policy-based scene classification into temporal scene detection, comprising the following steps:

[0076] Step A: Obtain the video, either from a live video stream or from a video file repository.

[0077] Step B: Extract frames from the acquired video at a certain frame rate and record the time of each frame in the video.

[0078] Step C: Perform scene recognition on each image from the extracted frames. The output is a scene classification prediction with confidence. Specifically, for each image, identify its scene category (e.g., classroom, playground, cafeteria, etc.); each predicted scene category corresponds to a confidence score, representing the model's certainty about the prediction (a floating-point number between 0 and 1). Specifically, when performing scene recognition on each image obtained from the extracted frames, a deep learning-based convolutional neural network (CNN) algorithm can be used. Relevant models include backbone networks such as ResNet and DenseNet. Pre-trained weights (e.g., models pre-trained on the Places365 dataset) can be loaded through transfer learning and fine-tuned for specific scene categories. For lightweight applications, lightweight CNN models such as MobileNet and ShuffleNet can be used to balance accuracy and inference speed. If capturing global semantic relationships in the image is required, a Transformer architecture (e.g., ViT) or a hybrid model combining attention mechanisms can also be used. The algorithm process is as follows: After preprocessing the image by adjusting its size and normalizing it, the image is input into the model to extract multi-level features. The scene category prediction and corresponding confidence score are output through a fully connected layer. Finally, the effective categories are selected according to the preset threshold.

[0079] Step D: Define the scene temporal detection result as a triple (scene_label, start_time, end_time), where scene_label is the scene label name, start_time is the start time, and end_time is the end time. For each scene category, if the confidence score of the image scene recognition is greater than a specified threshold, the image is considered to belong to that scene category, scene_label is defined as the scene category label, and the process proceeds to step E; otherwise, proceed to step F.

[0080] Step E: If there are no unfinished scene temporal detection results with the label scene_label (i.e., temporal detection results with end_time not set), then set the time t corresponding to the image to the start time start_time of the scene in the scene temporal detection results; otherwise, proceed to step F.

[0081] Step F: Define the maximum interval image count (max_age) for discontinuous scene recognition results, and the interval image count (age) for discontinuous scene recognition results. The initial value of age is 0. If an image does not belong to the scene category detected in the current temporal domain, the age count is incremented by 1; if the image belongs to the scene category detected in the current temporal domain, age is reset to 0. When age is greater than max_age, the scene is considered to have ended, and the time t corresponding to the last image belonging to that scene is set as the scene end time (end_time). At this point, a preliminary scene temporal domain detection result (scene_label, start_time, end_time) is obtained.

[0082] Step G: For the multiple images obtained in step B, execute steps C to F in chronological order to obtain multiple preliminary scene recognition results.

[0083] Step H: Define the minimum and maximum duration for each scene label, and filter the preliminary scene temporal detection results generated in Steps F and G, removing scene temporal detection results that are less than the minimum duration or greater than the maximum duration.

[0084] Step I: Use scene mutual exclusion filtering to filter the results of step G. When mutually exclusive scenes appear at the same time, only the detection results with high confidence threshold are retained (e.g., class scene and sleeping scene. When the sleeping scene and class scene are detected to appear at the same time, and the class scene has high confidence, only the class scene is retained).

[0085] Step J: Filter the results of Step I using scene time period reasonableness filtering. When the scene time domain detection result does not match the actual time period the scene should occur in, the scene time domain detection result is not retained (for example, kindergarten nap time is generally at noon, but if it is detected that napping occurs from 10:00 to 10:30 am, then the detection result is not retained). The final scene time domain detection result is obtained.

[0086] The advantage of this invention is that, compared with other time-domain conversion methods, the results obtained by this method are more accurate. This invention improves the accuracy of scene time-domain conversion by setting the maximum interval number of discontinuous scene recognition results, the minimum and maximum duration of time-domain detection results, scene mutual exclusion filtering, and scene time period rationality filtering.

[0087] This invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the methods described above.

[0088] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0089] The present invention also provides an electronic device. The electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method provided by the present invention.

[0090] The following is for reference. Figure 10 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing an electronic device according to embodiments of the present invention. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0091] like Figure 10 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the computer system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0092] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0093] In particular, according to the embodiments disclosed in this invention, the processes described in the above main step diagrams can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit 801, it performs the functions defined in the system of this invention.

[0094] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for converting scene classification into scene temporal detection, characterized in that, Includes the following steps: S110: Acquire video data, which comes from a real-time video stream or a stored video file repository; S120: The video data is processed by extracting frames according to a preset frame rate to obtain a series of static images arranged in chronological order, and the timestamp information of each image in the video is recorded. S130: Based on a deep learning model, scene category recognition is performed on each image obtained by frame extraction to obtain the scene category label and its confidence value corresponding to each image; S140: Based on the scene category label and the confidence value, and combined with the preset maximum interval number of discontinuous scene recognition results, generate preliminary scene temporal detection results; S150: Perform multi-level filtering optimization on the preliminary scene temporal detection results to obtain the final scene temporal detection results; Step S140 specifically includes: In the process of generating the preliminary scene temporal detection results, the maximum interval number of discontinuous scene recognition results (max_age) is defined, and the interval number of discontinuous scene recognition results (age) is initialized to 0; and the preliminary scene temporal detection results are defined as a triple (scene_label, start_time, end_time), where scene_label is the scene category label, start_time is the scene start time, and end_time is the scene end time. If the current image does not belong to the scene category detected in the current scene temporal domain, increment the age count by 1; When the current image belongs to the scene category detected in the temporal domain of the current scene, reset age to 0; When age exceeds max_age, the current scene is determined to end, and the timestamp of the last image belonging to the scene is recorded as the scene end time. At this time, a preliminary scene temporal detection result (scene_label, start_time, end_time) is obtained. Step S140 also includes the following steps: During the process of generating preliminary scene temporal detection results, for each scene category, if the scene recognition confidence value of the current image is higher than the preset threshold, the current image is marked as a valid frame of the scene category; otherwise, the current image is marked as an invalid frame, and subsequent judgment is made in combination with the maximum number of discontinuous images in the scene recognition results. Step S140 also includes the following steps: During the process of generating preliminary scene temporal detection results, for each scene category, when it is detected that the current image belongs to any scene category and there are no unfinished scene temporal detection results, the timestamp corresponding to the current image is marked as the start time of the scene.

2. The method as described in claim 1, characterized in that, Step S150 specifically includes: After generating preliminary scene temporal detection results, define the minimum duration threshold and the maximum duration threshold for each scene category; Scene time-domain segments that are less than the minimum duration threshold or greater than the maximum duration threshold in the preliminary scene time-domain detection results are removed.

3. The method as described in claim 2, characterized in that, Step S150 also includes the following steps: After generating preliminary scene temporal detection results, the preliminary scene temporal detection results are optimized using mutually exclusive scene filtering rules; When multiple mutually exclusive scenarios are detected within the same time period, only the scenario categories with confidence values ​​higher than the preset confidence threshold are retained as the final detection results.

4. The method as described in claim 3, characterized in that, Step S150 also includes the following steps: After generating preliminary scene temporal detection results, the preliminary scene temporal detection results are further optimized using the reasonableness constraints of scene time periods; For each detected scene time-domain segment, if its time range does not match the reasonable time period that the scene category should occur in actual application, then the scene time-domain segment is removed from the preliminary detection results.

5. The method as described in claim 4, characterized in that, Step S150 also includes the following steps: The final scene temporal detection result is output as a triple (scene_label, start_time, end_time), where scene_label represents the scene category label, start_time represents the scene start time, and end_time represents the scene end time.

6. A system for converting scene classification into scene temporal detection, characterized in that, The system performs the method for converting scene classification into scene temporal detection as described in any one of claims 1-5, the system comprising: The video acquisition module is used to acquire video data from real-time video streams or stored video file repositories; The frame extraction processing module is used to extract frames from video data according to a preset frame rate and record the timestamp information of each image. The scene recognition module is used to identify the scene category of each image obtained by frame extraction, and obtain the scene category label and its confidence value corresponding to each image; The temporal detection module is used to generate preliminary temporal detection results of the scene based on the scene category label and confidence value, combined with the preset maximum number of images at the interval of discontinuous scene recognition results; The result optimization module is used to perform multi-level filtering optimization on the preliminary scene temporal detection results to obtain the final scene temporal detection results.

7. The system as described in claim 6, characterized in that, The result optimization module specifically includes: The minimum and maximum duration filtering units are used to define the minimum and maximum duration thresholds for each scene category and to remove scene time-domain segments that are less than the minimum duration threshold or greater than the maximum duration threshold in the initial scene time-domain detection results.

Images