Method and system for converting scene classification into scene time domain detection

Abstract

The invention discloses a method and system for converting scene classification into scene time domain detection, and the method comprises the steps: obtaining video data from a real-time video stream or a stored video file warehouse; performing frame extraction processing on the video data according to a preset frame rate to obtain a series of static images arranged according to a time sequence, and recording timestamp information of each image in the video; performing scene category identification on each image obtained by frame extraction to obtain a scene category label corresponding to each image and a confidence value of the scene category label; according to the scene category label and the confidence value, generating a preliminary scene time domain detection result in combination with a preset maximum interval image number with a discontinuous scene recognition result; and performing multi-level filtering optimization on the preliminary scene time domain detection result to obtain a final scene time domain detection result. The accuracy of scene time domain detection can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and specifically provides a method and system for converting scene classification into scene time domain detection. Background Art

[0002] The current conversion of kindergarten indoor scene classification into scene time domain detection generally includes the following steps: (1) defining scene categories (such as kindergarten indoor scenes: classes, meals, group activities, and sleep) and training scene image classification models; (2) obtaining videos, which can be obtained from real-time video streams or from video file warehouses; (3) extracting frames from the video at a certain frame rate to obtain a series of static images sorted by time, and recording the time of each image in the video; (4) performing scene recognition on the images; (5) connecting the time of images with consistent scene recognition results in time sequence into time periods as the scene time domain detection results.

[0003] The conversion of existing scene classification into scene time domain detection has the following problems: due to errors in the scene recognition results of individual images, a scene detected in the time domain is interrupted into multiple scenes or recognized as an erroneous scene with a very short time, resulting in low accuracy of the scene time domain detection results. Summary of the Invention

[0004] The present invention provides a method and system for converting strategy-based scene classification into scene time domain detection. This method can solve the problem in the prior art of interrupting scene time domain detection results or generating erroneous scenes due to individual image recognition errors, and improve the accuracy and reliability of scene time domain detection.

[0005] In a first aspect, a method for converting a policy-based scene classification into a scene time domain detection is provided, comprising the following steps: the method first acquires video data, which can be extracted from a real-time video stream or from a stored video file repository; the video data is then subjected to frame extraction processing at a preset frame rate to obtain a series of static images arranged in chronological order, and the timestamp information of each image in the video is recorded; scene category recognition is then performed on each image obtained by frame extraction to obtain a scene category label and its confidence value corresponding to each image; a preliminary scene time domain detection result is generated based on the scene category label and confidence value, combined with a preset maximum number of discontinuous interval images for the scene recognition result; finally, a final scene time domain detection result is obtained by performing multi-level filtering optimization on the preliminary scene time domain detection result.

[0006] In the solution provided by this invention, by setting the maximum number of images between discontinuous scene recognition results, it is possible to effectively avoid interruptions in scene time-domain detection results due to individual image recognition errors. At the same time, by combining the minimum and maximum duration limits for scene time-domain detection, short-term or excessively long detection results that do not conform to actual scene logic are further eliminated. Furthermore, by introducing mutually exclusive scene filtering rules and scene time period rationality constraints, the accuracy and practicality of scene time-domain detection results can be further improved.

[0007] On the second aspect, a system for converting policy-based scene classification into scene time domain detection is provided, including: a video acquisition module for acquiring video data from a real-time video stream or a stored video file repository; a frame extraction processing module for extracting frames from the video data at a preset frame rate and recording the timestamp information of each image; a scene recognition module for performing scene category recognition on each image obtained by frame extraction, and obtaining the scene category label and its confidence value corresponding to each image; a time domain detection module for generating preliminary scene time domain detection results based on the scene category label and confidence value, combined with a preset maximum number of discontinuous interval images of the scene recognition results; and a result optimization module for performing multi-level filtering optimization on the preliminary scene time domain detection results to obtain the final scene time domain detection results.

[0008] Through the above technical solution, the present invention provides a method and system for converting strategy-based scene classification into scene time domain detection, which can effectively solve the problem in the prior art that scene time domain detection results are interrupted or erroneous scenes are generated due to individual image recognition errors, and improve the accuracy and reliability of scene time domain detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of the overall architecture of a system for converting policy-based scene classification into scene time-domain detection provided by an embodiment of the present invention;

[0010] Figure 2 This is a schematic diagram of the workflow of the frame extraction processing module in an embodiment of the present invention, showing the process of video data frame extraction and timestamp recording;

[0011] Figure 3 A schematic diagram of the working principle of the scene recognition module in an embodiment of the present invention, including the generation process of scene category labels and confidence values;

[0012] Figure 4 This is a schematic diagram of the logical structure of the time domain detection module in an embodiment of the present invention, showing the process of generating the preliminary scene time domain detection results;

[0013] Figure 51 is a schematic diagram of the operation flow of the scene continuity determination unit in an embodiment of the present invention, describing the influence of the maximum number of interval images on the scene continuity determination;

[0014] Figure 6 This is a schematic diagram of the working mechanism of the mutually exclusive scene filtering unit in an embodiment of the present invention, illustrating how to filter mutually exclusive scenes by confidence value;

[0015] Figure 7 This is a functional diagram of a time period rationality filtering unit in an embodiment of the present invention, illustrating how to remove scene segments that do not meet the actual time period constraints;

[0016] Figure 8 A schematic diagram of the overall process of converting the policy-based scene classification into the scene time domain detection method provided by an embodiment of the present invention;

[0017] Figure 9 A schematic diagram of a specific process of converting the policy-based scene classification into scene time domain detection method provided by an embodiment of the present invention;

[0018] Figure 10 A functional block diagram of a computer system provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0019] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] Example 1

[0021] The embodiment of the present invention provides a method and system for converting scene classification into scene time domain detection based on strategy, which aims to solve the problem of interruption of scene time domain detection results or generation of erroneous scenes due to individual image recognition errors through a multi-level filtering optimization mechanism. Figures 1 to 8 The specific embodiments of the present invention are described in detail with reference to the accompanying drawings.

[0022] In practical applications, the system of the embodiment of the present invention is composed of multiple modules working together to complete the scene time domain detection task. First, the video acquisition module is used to obtain video data from the real-time video stream or the stored video file warehouse. The video acquisition module is connected to the frame extraction processing module and transmits the obtained video data to the frame extraction processing module for subsequent processing. Figure 2As shown, the frame extraction module extracts frames from the video data at a preset frame rate and records the timestamp information for each image. The frame extraction module contains a timer unit that accurately records the time position of each frame in the video and stores the timestamp information in a one-to-one correspondence with the image. The output of the frame extraction module is directly passed to the scene recognition module for further analysis.

[0023] The scene recognition module receives the image data from the frame extraction processing module and performs scene category recognition on it. Figure 3 As shown, the scene recognition module includes a deep learning model that extracts and classifies input images to generate a corresponding scene category label and confidence score for each image. The scene recognition module also includes a confidence screening unit, which performs preliminary screening of recognition results. When the scene recognition confidence score of an image exceeds a preset threshold, the image is marked as a valid frame; otherwise, it is marked as an invalid frame. The output of the confidence screening unit, along with the original recognition result, is passed to the time domain detection module.

[0024] The main function of the time domain detection module is to generate preliminary scene time domain detection results based on scene category labels and confidence values. Figure 4 As shown, the internal structure of the time domain detection module

[0025] It includes a scene continuity judgment unit and a scene start time marking unit. The scene continuity judgment unit defines the maximum number of interval images (max_age) for discontinuous scene recognition results, and initializes the number of interval images (age) for discontinuous scene recognition results to 0. When the current image does not belong to the current scene category, the scene continuity judgment unit adds 1 to the age count; when the current image belongs to the current scene category, age is reset to 0. If age exceeds max_age, the current scene is determined to be over, and the timestamp corresponding to the last image belonging to the scene is recorded as the scene end time. The scene start time marking unit is used to mark the timestamp corresponding to the image as the start time of each scene category when it is detected that the current image belongs to any scene category and there is no unfinished time domain detection result for the scene.

[0026] The preliminary scene time domain detection results are then passed to the result optimization module for multi-level filtering optimization. The result optimization module includes a minimum duration and maximum duration filtering unit, a mutually exclusive scene filtering unit, and a time period rationality filtering unit. The minimum duration and maximum duration filtering unit is used to define the minimum duration threshold and maximum duration threshold for each scene category, and remove scene time domain segments that are less than the minimum duration threshold or greater than the maximum duration threshold in the preliminary scene time domain detection results. The working principle of the mutually exclusive scene filtering unit is as follows: Figure 6As shown in the figure, its processing logic is that when multiple mutually exclusive scenes are detected in the same time period, only the scene categories with confidence values ​​higher than the preset confidence threshold are retained as the final detection results. For example, the "class" and "sleep" scenes in a kindergarten indoor scene usually do not occur at the same time. Therefore, when these two scenes are detected to appear in the same time period, the mutually exclusive scene filtering unit will give priority to the scene category with a higher confidence value. The function of the time period rationality filtering unit is as follows Figure 7 As shown in Figure 2, its function is to remove scene segments that do not meet the actual time period constraints. For example, the "sleeping" scene in the kindergarten indoor scene usually occurs during the noon period. If a "sleeping" scene is detected between 10:00 and 10:30 in the morning, the time period rationality filtering unit will treat it as an abnormal result and remove it.

[0027] like Figure 1 As shown in Figure 1, the various modules and units described above are connected via a data bus, forming a complete system architecture. The video acquisition module transmits video data to the frame extraction processing module, which then passes the extracted image data to the scene recognition module. The scene recognition module then passes the recognition results to the time domain detection module. The time domain detection module generates preliminary scene time domain detection results and passes them to the result optimization module. The result optimization module ultimately outputs the scene time domain detection results in the form of a triplet (scene_label, start_time, end_time).

[0028] In practical application scenarios, the embodiments of the present invention can be applied to monitoring systems for indoor scenes in kindergartens. For example, the system acquires video data from a real-time surveillance camera through a video acquisition module. The frame extraction processing module extracts the video at a frame rate of 10 frames per second and records the timestamp information of each frame. The scene recognition module performs scene category identification on the extracted images and generates a scene category label and its confidence value corresponding to each image. Suppose the image recognition results within a certain period of time show that the "class" scene appears for 50 consecutive frames and the "sleep" scene appears for 10 consecutive frames, and the confidence value of the "class" scene is higher than that of the "sleep" scene. The time domain detection module sets max_age to 15 frames based on the logic of the scene continuity judgment unit, and combines the threshold settings of the minimum and maximum duration filter units to generate preliminary scene time domain detection results. The result optimization module optimizes the preliminary results through the mutually exclusive scene filtering unit and the time period rationality filtering unit, and ultimately outputs the scene time domain detection results in the form of triples. For example, the output result may be (class, 09:00:00, 09:10:00), which means that the "class" scene lasts from 09:00:00 to 09:10:00.

[0029] The embodiment of the present invention improves the accuracy and reliability of scene time domain detection through a multi-level filtering optimization mechanism.

[0030] The following further explains the specific implementation principle of the present invention in conjunction with a specific application scenario.

[0031] In the monitoring system of the kindergarten indoor scene, the operation process of the embodiment of the present invention can be decomposed into multiple steps for detailed description. First, the video acquisition module collects video data through the real-time monitoring camera and transmits the video stream to the frame extraction processing module. The frame extraction processing module extracts the video data according to the preset frame rate (for example, 10 frames per second) and records the timestamp information of each image. Figure 2 As shown in the figure, the timer unit inside the frame extraction processing module accurately records the time position of each frame image in the video and stores the timestamp and image in a one-to-one correspondence. This process ensures the temporal correlation of subsequent scene recognition and time domain detection.

[0032] Subsequently, the frame extraction processing module passes the extracted image data to the scene recognition module. The scene recognition module uses its internal deep learning model to perform feature extraction and classification operations on each image, generating the corresponding scene category label and its confidence value. Figure 3 As shown in Figure 1, the confidence screening unit performs a preliminary screening of the generated recognition results, retaining only valid frames with confidence values ​​above a preset threshold. This mechanism effectively reduces the interference of low-confidence frames on subsequent detection results, thereby improving the reliability of scene recognition.

[0033] Next, the time domain detection module generates preliminary scene time domain detection results based on the scene category label and confidence value. Figure 4 As shown, the scene continuity judgment unit defines the maximum number of images (max_age) between which scene recognition results are discontinuous, and initializes the number of images (age) to 0. When the current image does not belong to the current scene category, the scene continuity judgment unit increments the age counter by 1; when the current image belongs to the current scene category, the age counter is reset to 0. If age exceeds max_age, the current scene is considered to have ended, and the timestamp corresponding to the last image belonging to the scene is recorded as the scene end time. This logical design avoids interruptions in scene temporal detection results due to individual image recognition errors, thereby ensuring the consistency of detection results.

[0034] At the same time, the minimum and maximum duration filtering units further optimize the initial scene temporal detection results. This unit defines minimum and maximum duration thresholds for each scene category and removes scene temporal segments that do not meet these thresholds. For example, if a scene segment's duration is less than the minimum duration threshold or greater than the maximum duration threshold, it is considered an anomaly and removed. This mechanism effectively removes short or excessively long scene segments caused by misidentification, thereby improving the accuracy of detection results.

[0035] After the preliminary scene time domain detection results are generated, the result optimization module performs multi-level filtering optimization on them. The working principle of the mutually exclusive scene filtering unit is as follows: Figure 6 As shown in the figure, when multiple mutually exclusive scenes are detected within the same time period, only the scene category with the higher confidence value is retained as the final detection result. For example, in a kindergarten indoor scene, the "class" and "sleep" scenes usually do not occur at the same time. Therefore, when these two scenes appear in the same time period, the mutually exclusive scene filtering unit prioritizes the scene category with the higher confidence value. This design solves the problem of multiple scene overlap caused by scene category confusion and further improves the rationality of the detection results.

[0036] In addition, the function of the time period rationality filtering unit is as follows Figure 7 As shown in the figure, its function is to eliminate scene segments that do not conform to the actual time period constraints. For example, "sleeping" scenes in kindergarten indoor scenes usually occur during the afternoon. If a "sleeping" scene is detected between 10:00 and 10:30 in the morning, the time period rationality filtering unit will treat it as an anomaly and eliminate it. This mechanism effectively eliminates unreasonable scene time domain segments caused by misidentification, thereby improving the credibility of the detection results.

[0037] Finally, the result optimization module outputs the scene temporal detection results after multi-level filtering and optimization as a triplet (scene_label, start_time, end_time). The various modules and units of the entire system are connected via a data bus, forming a complete automated processing flow. From video data acquisition to final result output, the system achieves efficient and accurate scene temporal detection.

[0038] The above steps demonstrate that the present invention improves the accuracy and reliability of scene temporal detection through a multi-layered filtering and optimization mechanism. In particular, in kindergarten indoor scene applications, the system effectively addresses issues such as interrupted scene temporal detection results or incorrect scene generation due to individual image recognition errors, thus meeting practical application requirements.

[0039] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the method embodiment, which will not be repeated here.

[0040] Example 2

[0041] like Figure 8 As shown, a method for converting strategy-based scene classification into scene time domain detection is provided, including the following steps:

[0042] S110: The method first obtains video data, which can be extracted from a real-time video stream or a stored video file repository;

[0043] S120: Then, the video data is subjected to frame extraction processing according to a preset frame rate to obtain a series of static images arranged in chronological order, and the timestamp information of each image in the video is recorded;

[0044] S130: Then, scene category recognition is performed on each image obtained by extracting frames to obtain a scene category label and a confidence value corresponding to each image;

[0045] S140: Then, based on the scene category label and the confidence value, combined with the preset maximum number of discontinuous images of the scene recognition results, a preliminary scene time domain detection result is generated;

[0046] S150: Finally, the preliminary scene time domain detection result is subjected to multi-level filtering optimization to obtain the final scene time domain detection result.

[0047] In an embodiment of the present invention, by setting the maximum number of images between discontinuous scene recognition results, it is possible to effectively avoid interruptions in scene time domain detection results due to individual image recognition errors. At the same time, by combining the minimum and maximum duration limits for scene time domain detection, short-term or excessively long detection results that do not conform to actual scene logic are further eliminated. Furthermore, by introducing mutually exclusive scene filtering rules and scene time period rationality constraints, the accuracy and practicality of scene time domain detection results can be further improved.

[0048] In some embodiments, step S140 specifically includes:

[0049] In the process of generating the preliminary scene time domain detection results, the maximum number of interval images (max_age) for discontinuous scene recognition results is defined, and the number of interval images (age) for discontinuous scene recognition results is initialized to 0;

[0050] When the current image does not belong to the current scene category, the age count is increased by 1;

[0051] When the current image belongs to the current scene category, reset age to 0;

[0052] When age exceeds max_age, the current scene is determined to be over, and the timestamp corresponding to the last image belonging to the scene is recorded as the scene end time.

[0053] Step S140 constructs a buffer mechanism for scene temporal continuity by defining the maximum number of discontinuous images (max_age) between scene recognition results and initializing the interval count (age): when the current image does not belong to the current scene category, age is gradually accumulated, and when it does, the count is reset. The scene is determined to be over only when the number of consecutive non-target scene images exceeds max_age. This mechanism effectively filters out the problem of scene interruption errors caused by image recognition errors (such as short-term interference and noise) in a single frame or a few frames, allowing the scene temporal detection results to tolerate a certain degree of recognition discontinuity, thereby enhancing the robustness of the detection results to accidental recognition errors, ensuring that the actual continuous scene will not be incorrectly segmented due to the misjudgment of individual images, improving the accuracy and stability of the scene temporal boundary division, and more in line with the continuous characteristics of real scenes in the time dimension.

[0054] In some embodiments, step S140 further includes the following steps:

[0055] In the process of generating preliminary scene time domain detection results, for each scene category, when the scene recognition confidence value of the current image is higher than the preset threshold, the current image is marked as a valid frame of the scene category to which it belongs; otherwise, the current image is marked as an invalid frame, and subsequent judgment is made in combination with the maximum number of discontinuous interval images of the scene recognition results.

[0056] In some embodiments, step S140 further includes the following steps:

[0057] In the process of generating preliminary scene time domain detection results, for each scene category, when it is detected that the current image belongs to any scene category and there is no unfinished time domain detection result of the scene, the timestamp corresponding to the current image is marked as the start time of the scene.

[0058] This technical solution ensures the uniqueness and accuracy of the starting point of the scene time domain segment through the rule of "marking the start time only when the current image belongs to a certain scene category and there is no unfinished time domain detection result of the scene". On the one hand, it avoids the redundant problem of repeatedly marking the starting point of the same scene before it ends, and on the other hand, it ensures that the start of each new scene segment is based on the real scene switching boundary (rather than the accidental recognition fluctuation of the intermediate frame). This mechanism enables the generated preliminary scene time domain detection results to have a clear segmentation logic in the time dimension, which is consistent with the natural characteristics of continuous and non-overlapping actual scenes, and provides a reliable time benchmark for subsequent duration calculation, mutually exclusive filtering and other steps, thereby improving the structural and logical consistency of the overall detection results.

[0059] In some embodiments, step S150 specifically includes:

[0060] After generating the preliminary scene time domain detection results, define the minimum duration threshold and maximum duration threshold for each scene category;

[0061] The scene time domain segments whose duration is less than a minimum duration threshold or greater than a maximum duration threshold in the preliminary scene time domain detection result are eliminated.

[0062] By defining the minimum and maximum duration thresholds for scene categories and eliminating non-compliant time domain segments, this technical solution can effectively filter out short-term invalid scenes caused by accidental recognition errors (such as instantaneous scenes caused by single-frame misdetection) and extremely long abnormal scenes that violate objective laws (such as unreasonable scenes that last for several hours), making the detection results more consistent with the logical characteristics of real scenes in the time dimension. This operation not only reduces false detections and redundant information, and improves the rationality and credibility of scene time domain detection results, but also lays a solid foundation for subsequent steps such as mutually exclusive scene filtering and time period rationality analysis, ultimately making the output scene time domain segments more in line with actual application needs (such as surveillance video analysis, intelligent video retrieval, etc.).

[0063] In some embodiments, step S150 further includes the following steps:

[0064] After generating preliminary scene time domain detection results, optimizing the preliminary scene time domain detection results using mutually exclusive scene filtering rules;

[0065] When multiple mutually exclusive scenes are detected in the same time period, only the scene categories with confidence values ​​higher than the preset confidence threshold are retained as the final detection results.

[0066] This technical solution can effectively solve the logical contradiction problem of the coexistence of conflicting scenes in the same time period by introducing mutually exclusive scene filtering rules and combining them with confidence threshold screening. Specifically, when mutually exclusive scenes (such as "class" and "sleep", "meeting" and "idle") are detected to exist at the same time in the same time period, by eliminating scene categories with confidence levels lower than the threshold and retaining only results with higher credibility, it can ensure that the final detection results are consistent with the logical consistency of the real scene (for example, there can only be one main activity scene in the classroom at the same time). This operation not only eliminates the semantic conflicts in the detection results and improves the accuracy and reliability of scene labels, but also provides a purer data source for subsequent applications such as video content analysis and event association judgment, thereby enhancing the usability of the entire scene time domain detection system in actual business and reducing the risk of misjudgment due to logical contradictions.

[0067] In some embodiments, step S150 further includes the following steps:

[0068] After generating the preliminary scene time domain detection results, the preliminary scene time domain detection results are further optimized using the scene time period rationality constraint conditions;

[0069] For each detected scene time domain segment, if its time range does not match the reasonable time period that the scene category should occur in actual applications, the scene time domain segment will be eliminated from the preliminary detection results.

[0070] This technical solution optimizes the preliminary scene time domain detection results by utilizing the rationality constraints of the scene time period, matches the time range of the detected scene time domain fragments with the reasonable time period in which the scene should occur in actual applications, and eliminates fragments that do not match the time range, thereby ensuring that the detection results conform to the real logic and objective laws in the time dimension (for example, avoiding lunch break scenes appearing in the morning and commuting scenes appearing at night, etc.), effectively improving the rationality, credibility and practicality of the scene time domain detection results, and providing data support that is more in line with actual needs for subsequent applications such as intelligent scheduling, behavior analysis, and abnormal warning. At the same time, it reduces the risk of misjudgment due to time logic contradictions and enhances the interpretability and business adaptability of the detection results.

[0071] In some embodiments, step S150 further includes the following steps:

[0072] The final scene time domain detection result is output as a triplet (scene_label, start_time, end_time), where scene_label represents the scene category label, start_time represents the scene start time, and end_time represents the scene end time.

[0073] In an embodiment of the present invention, by outputting the scene time domain detection results in the form of triples, the category, start time and end time of each scene time domain segment can be intuitively displayed, which is convenient for subsequent scene analysis and application.

[0074] Example 3

[0075] like Figure 9 As shown, the present invention aims at the above problems and provides a method for converting scene classification based on policy into scene time domain detection. To solve the above technical problems, the technical solution provided by the present invention is a method for converting scene classification based on policy into scene time domain detection, comprising the following steps:

[0076] Step A: Obtain the video. The video can be obtained from a real-time video stream or from a video file repository.

[0077] Step B: extract frames from the acquired video at a certain frame rate and record the time of each frame image in the video.

[0078] Step C: Perform scene recognition on each image mentioned in the frame extraction, and the output result is a scene classification prediction with confidence. Specifically, for each image, identify the scene category to which it belongs (for example, classroom, playground, cafeteria, etc.); each predicted scene category corresponds to a confidence score, which indicates the degree of certainty of the model about the prediction (a floating point number between 0 and 1). Specifically, when performing scene recognition on each image obtained by frame extraction, a convolutional neural network (CNN) algorithm based on deep learning can be used. Related models include backbone networks such as ResNet and DenseNet. Pre-trained weights can be loaded through transfer learning (for example, models pre-trained on the Places365 dataset) and fine-tuned for specific scene categories. For lightweight application scenarios, lightweight CNN models such as MobileNet and ShuffleNet can be used to balance accuracy and inference speed. If you need to capture the global semantic relationship of the image, you can also use a Transformer architecture (such as ViT) or a hybrid model combined with an attention mechanism. The algorithm process is as follows: after preprocessing the image by resizing and normalizing it, the model is input to extract multi-level features, and the scene category prediction and corresponding confidence score are output through the fully connected layer. Finally, the valid category is screened according to the preset threshold.

[0079] Step D: Define the scene time domain detection result as a triplet (scene_label, start_time, end_time), where scene_label is the scene label name, start_time is the start time, and end_time is the end time. For each scene category, if the confidence level of the image scene recognition is greater than the specified threshold, the image is considered to belong to that scene category, scene_label is defined as the scene category label, and the process proceeds to Step E; otherwise, the process proceeds to Step F.

[0080] Step E: If there is no unfinished scene time domain detection result with the label scene_label (i.e., the time domain detection result with unset end_time), then the time t corresponding to the image is set to the start time start_time of the scene in the scene time domain detection result; otherwise, proceed to step F.

[0081] Step F: Define the maximum number of images (max_age) between which scene recognition results are discontinuous, and the initial value of age (age) for discontinuous scene recognition results. If the image does not belong to the scene category detected by the current scene time domain, the age count is incremented by 1; if the image belongs to the scene category detected by the current scene time domain, age is reset to 0. When age is greater than max_age, the scene is considered to have ended, and the time t corresponding to the last image belonging to the scene is set as the end time of the scene (end_time). At this point, a preliminary scene time domain detection result (scene_label, start_time, end_time) is obtained.

[0082] Step G: cyclically execute steps C to F for the multiple images obtained in step B in chronological order to obtain multiple preliminary scene recognition results.

[0083] Step H: define the minimum and maximum durations of each scene label, filter the preliminary scene time domain detection results generated in steps F and G, and filter out scene time domain detection results that are less than the minimum duration and greater than the maximum duration.

[0084] Step I: Use scene mutual exclusion filtering to filter the results of step G. When mutually exclusive scenes appear at the same time, only the detection results with a high confidence threshold are retained (for example, a class scene and a sleeping scene. When it is detected that a sleeping scene and a class scene appear at the same time, and the confidence of the class scene is high, only the class scene is retained).

[0085] Step J: Filter the results of step I using the reasonableness filter of the scene time period. If the scene time domain detection result does not match the time period in which the scene should actually occur, the scene time domain detection result is not retained (for example, kindergarten students generally sleep at noon, and if it is detected that they sleep between 10:00 and 10:30 in the morning, the detection result is not retained). The final scene time domain detection result is obtained.

[0086] The advantages of the embodiments of the present invention are that compared with other time domain conversion methods, this method has higher accuracy. The accuracy of scene time domain conversion is improved by setting the maximum number of images between discontinuous scene recognition results, the minimum and maximum durations of time domain detection results, scene mutual exclusion filtering, and scene time period rationality filtering.

[0087] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements any one of the above methods when executed by a processor.

[0088] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. Of course, there are other ways of readable storage media, such as quantum memory, graphene memory, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0089] The present invention also provides an electronic device. The electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the present invention.

[0090] Reference below Figure 10 , which shows a schematic structural diagram of a computer system 800 of an electronic device suitable for implementing an embodiment of the present invention. Figure 10 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0091] like Figure 10 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the computer system 800 are also stored in the RAM 803. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0092] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, and the like; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 810 as needed, so that computer programs read therefrom can be installed in the storage section 808 as needed.

[0093] In particular, according to embodiments disclosed herein, the processes described in the main step diagrams above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via the communication section 809 and / or installed from removable media 811. When the computer program is executed by the central processing unit 801, the above-described functions defined in the system of the present invention are performed.

[0094] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for converting scene classification into scene time domain detection, characterized in that: The following steps are involved: S110: Acquire video data, where the video data comes from a real-time video stream or a stored video file repository; S120: performing frame extraction processing on the video data at a preset frame rate to obtain a series of static images arranged in chronological order, and recording timestamp information of each image in the video; S130: Perform scene category recognition on each image obtained by extracting frames to obtain a scene category label and a confidence value corresponding to each image; S140: generating a preliminary scene time domain detection result based on the scene category label and the confidence value, combined with a preset maximum number of discontinuous images of the scene recognition result; S150: Perform multi-level filtering optimization on the preliminary scene time domain detection result to obtain a final scene time domain detection result.

2. The method according to claim 1, wherein Step S140 specifically includes: In the process of generating the preliminary scene time domain detection results, the maximum number of interval images (max_age) for discontinuous scene recognition results is defined, and the number of interval images (age) for discontinuous scene recognition results is initialized to 0; When the current image does not belong to the current scene category, the age count is increased by 1; When the current image belongs to the current scene category, reset age to 0; When age exceeds max_age, the current scene is determined to be over, and the timestamp corresponding to the last image belonging to the scene is recorded as the scene end time.

3. The method according to claim 2, wherein Step S140 further includes the following steps: In the process of generating preliminary scene time domain detection results, for each scene category, when the scene recognition confidence value of the current image is higher than the preset threshold, the current image is marked as a valid frame of the scene category to which it belongs; otherwise, the current image is marked as an invalid frame, and subsequent judgment is made in combination with the maximum number of discontinuous interval images of the scene recognition results.

4. The method according to claim 3, wherein Step S140 further includes the following steps: In the process of generating preliminary scene time domain detection results, for each scene category, when it is detected that the current image belongs to any scene category and there is no unfinished time domain detection result of the scene, the timestamp corresponding to the current image is marked as the start time of the scene.

5. The method according to claim 1, wherein Step S150 specifically includes: After generating the preliminary scene time domain detection results, define the minimum duration threshold and maximum duration threshold for each scene category; The scene time domain segments whose duration is less than a minimum duration threshold or greater than a maximum duration threshold in the preliminary scene time domain detection result are eliminated.

6. The method according to claim 5, wherein: Step S150 further includes the following steps: After generating preliminary scene time domain detection results, optimizing the preliminary scene time domain detection results using mutually exclusive scene filtering rules; When multiple mutually exclusive scenes are detected in the same time period, only the scene categories with confidence values ​​higher than the preset confidence threshold are retained as the final detection results.

7. The method according to claim 6, wherein Step S150 further includes the following steps: After generating the preliminary scene time domain detection results, the preliminary scene time domain detection results are further optimized using the scene time period rationality constraint conditions; For each detected scene time domain segment, if its time range does not match the reasonable time period that the scene category should occur in actual applications, the scene time domain segment will be eliminated from the preliminary detection results.

8. The method according to claim 7, wherein Step S150 further includes the following steps: The final scene time domain detection result is output as a triplet (scene_label, start_time, end_time), where scene_label represents the scene category label, start_time represents the scene start time, and end_time represents the scene end time.

9. A system for converting scene classification into scene time domain detection, characterized in that: include: Video acquisition module, used to obtain video data from real-time video streams or stored video file warehouses; The frame extraction processing module is used to extract frames from the video data according to a preset frame rate and record the timestamp information of each image; The scene recognition module is used to identify the scene category of each image obtained by frame extraction and obtain the scene category label and confidence value corresponding to each image; The time domain detection module is used to generate preliminary scene time domain detection results based on the scene category label and confidence value, combined with the preset maximum number of images with discontinuous scene recognition results; The result optimization module is used to perform multi-level filtering optimization on the preliminary scene time domain detection result to obtain the final scene time domain detection result.

10. The system according to claim 9, wherein: The result optimization module specifically includes: The minimum duration and maximum duration filtering unit is used to define the minimum duration threshold and maximum duration threshold for each scene category, and to remove scene time domain segments that are less than the minimum duration threshold or greater than the maximum duration threshold in the preliminary scene time domain detection results.

Images