Index information generation method and device, video retrieval method and device and electronic equipment
By using dynamic sampling and multimodal large model analysis, key video frames are selected and index information is generated, solving the problem of text extraction from moving objects in security detection scenarios. This achieves efficient text recognition and video frame localization, improving the user experience.
Patent Information
- Application Number
- CN202512016906.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-14
AI Technical Summary
In security inspection scenarios, existing technologies struggle to effectively extract text information from moving objects, especially when dealing with vehicles of different brands, regions, and shapes. OCR and deep learning algorithms have low recognition rates and struggle to locate video frames containing text, impacting the user experience.
Multiple sampled video frames are acquired through dynamic sampling. Key video frames are analyzed using a multimodal large model to filter out object description information and object-related text information, and generate index information. This process includes steps such as edge detection, grayscale statistics, and clustering, which reduces the workload of subsequent multimodal large models and improves text recognition accuracy.
It achieves accurate text extraction and video frame localization of moving objects in security detection scenarios, improves user experience, reduces computational load, and ensures the accuracy of text recognition and extraction.
Smart Images

Figure CN121858773A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of video information extraction and retrieval technology, and more specifically, to an index information generation method, a video retrieval method, an apparatus, and an electronic device. Background Technology
[0002] With the continuous development of computer technology, computer vision can be used to extract text from various types of information. For example, Optical Character Recognition (OCR) can be used to convert printed or handwritten text in images into editable electronic text. Alternatively, deep learning algorithms can be used to extract text from images, such as Convolutional Neural Networks (CNN), Long Short-Term Memory networks (LSTM), and Convolutional Recurrent Neural Networks (CRNN).
[0003] However, in security detection scenarios, the objects being detected are typically moving, such as people and vehicles. Furthermore, the brands, regions, and shapes of these objects vary considerably. Therefore, OCR or deep learning algorithms are less effective at extracting text from videos. Additionally, during later video playback, it is difficult to locate video frames containing text using these methods, impacting the user experience. Summary of the Invention
[0004] In view of this, the present disclosure provides an index information generation method, a video retrieval method, an apparatus, and an electronic device.
[0005] One aspect of this disclosure provides a method for generating index information, comprising: dynamically sampling a detection video based on its video content to obtain multiple sampled video frames, wherein the sampled video frames include a target object; filtering multiple key video frames from the multiple sampled video frames based on object information of the target object in the multiple sampled video frames; analyzing the multiple key video frames separately using a multimodal large model to obtain object description information and object-related text information corresponding to each key video frame; and generating index information for the detection video based on a multimodal information set composed of the multiple key video frames, the object description information corresponding to each key video frame, and the object-related text information, so as to determine the information set based on the retrieval information and the index information when receiving retrieval information for the detection video.
[0006] According to embodiments of this disclosure, a multimodal large model is used to analyze multiple key video frames to obtain object description information and object-related text information corresponding to each key video frame. This includes: performing semantic analysis on the target objects in each key video frame using the multimodal large model to obtain object description information; and performing text recognition on the target objects using the multimodal large model to obtain object-related text information.
[0007] According to embodiments of this disclosure, multiple key video frames are selected from multiple sampled video frames based on object information of a target object in multiple sampled video frames, including: performing edge detection on the target object region of each sampled video frame to obtain edge detection results; selecting multiple intermediate video frames from multiple sampled video frames based on the edge detection results, wherein the multiple intermediate video frames are arranged in chronological order; and selecting multiple key video frames from multiple intermediate video frames based on object information of the target object in the multiple intermediate video frames.
[0008] According to embodiments of this disclosure, the object information includes: a spliced vector obtained by concatenating edge feature vectors and grayscale statistical vectors, wherein the edge feature vectors include the coordinates of multiple edge points obtained after edge detection of the target object region, and the grayscale statistical vectors include multiple grayscale index values for the target object region; based on the object information for the target object in multiple intermediate video frames, multiple key video frames are selected from the multiple intermediate video frames, including: dividing the multiple intermediate video frames into multiple first video segments according to time order; and selecting key video frames from each first video segment based on the spliced vectors of the multiple intermediate video frames in each first video segment.
[0009] According to embodiments of this disclosure, key video frames are selected from each first video segment based on the splicing vectors of multiple intermediate video frames in each first video segment, including: for each first video segment, determining the cluster center of the first video segment based on the valid information of the first intermediate video frame in the first video segment; clustering the splicing vectors of each of the multiple intermediate video frames based on the cluster center to obtain a clustering result; determining the target splicing vector that satisfies the clustering condition from the multiple splicing vectors based on the clustering result; and using the intermediate video frame corresponding to the target splicing vector as the key video frame for the first video segment.
[0010] According to embodiments of this disclosure, the object information includes noise information of the target object region; multiple intermediate video frames are arranged in chronological order; multiple key video frames are obtained by filtering from the multiple intermediate video frames based on the object information of the target object in the multiple intermediate video frames, including: deleting intermediate video frames with noise information greater than a noise threshold from the multiple intermediate video frames based on the noise information of the target object region in each intermediate video frame, to obtain updated multiple intermediate video frames; and obtaining multiple key video frames by filtering from the updated multiple intermediate video frames.
[0011] According to embodiments of this disclosure, the object information includes the location information of the target object region; based on the object information of the target object in multiple intermediate video frames, multiple key video frames are selected from the multiple intermediate video frames, including: determining the motion change information of the target object between multiple intermediate video frames based on the location information of the target object region in each intermediate video frame; and selecting multiple key video frames from the multiple intermediate video frames based on the motion change information.
[0012] According to embodiments of this disclosure, multiple key video frames are selected from multiple intermediate video frames based on motion change information, including: if the motion change information of the target object between two adjacent intermediate video frames satisfies the motion difference condition, a key video frame is determined from the two adjacent intermediate video frames.
[0013] According to embodiments of this disclosure, multiple key video frames are selected from multiple intermediate video frames based on motion change information, including: taking an intermediate video frame that satisfies a first condition as the first frame; determining multiple intermediate video frames adjacent to the intermediate video frame that satisfies the first condition from the multiple intermediate video frames to obtain a second video segment; determining cumulative motion change information based on the motion change information of the target object in the second video segment; and determining at least one key video frame from the second video segment using a selection strategy that matches the cumulative motion change information.
[0014] According to embodiments of this disclosure, dynamic sampling of the detection video based on its content yields multiple sampled video frames. This includes: performing object recognition on a fixed region in the detection video to obtain a first detection result, wherein the fixed region corresponds to the detection location targeted by the detection video; performing quality detection and illumination detection on the target object region of the detection video to obtain a second and a third detection result, wherein the target object region is a region including the target object; performing inter-frame structural similarity detection on the detection video to obtain a fourth detection result; determining multiple sampling frequencies for the detection video based on at least one of the first, second, third, and fourth detection results; and sampling the detection video using the multiple sampling frequencies to obtain multiple sampled video frames.
[0015] According to embodiments of this disclosure, determining multiple sampling frequencies for a detection video based on at least one of a first detection result, a second detection result, a third detection result, and a fourth detection result includes: determining a first sampling frequency when a third video segment in the detection video meets a second condition, wherein the second condition includes: the first detection result indicates that the target object is located within a fixed area, the second detection result indicates that the target object area meets a quality condition, and the fourth detection result indicates that the inter-frame structural similarity of the third video segment is greater than a first threshold; determining a second sampling frequency when a fourth video segment in the detection video meets a third condition, wherein the third condition includes one of the following: the first detection result indicates that a portion of the target object is located within a fixed area, the second detection result indicates that the target object area meets a quality condition, and the fourth detection result indicates that the inter-frame structural similarity of the fourth video segment is greater than a second threshold and less than a first threshold; determining a third sampling frequency when a fifth video segment in the detection video meets a fourth condition, wherein the third condition includes one of the following: the second detection result indicates that the target object area does not meet a quality condition, and the third detection result indicates that the target object area does not meet an illumination condition; the frequency values of the first sampling frequency, the second sampling frequency, and the third sampling frequency increase sequentially.
[0016] Another aspect of this disclosure provides a video retrieval method, comprising responding to receiving retrieval information from a user, wherein the retrieval information includes at least one of: a first sub-retrieval information of an image modality and a second sub-retrieval information of a text modality; determining target index information matching the retrieval information from a plurality of candidate index information to obtain a target detection video indicated by the target index information, wherein the candidate index information is generated according to the above-described index information generation method.
[0017] Another aspect of this disclosure provides an index information generation apparatus, comprising: a sampling module for dynamically sampling a detection video based on its video content to obtain multiple sampled video frames, wherein the sampled video frames include a target object; a filtering module for filtering multiple key video frames from the multiple sampled video frames based on object information of the target object in the multiple sampled video frames; an analysis module for analyzing the multiple key video frames separately using a multimodal large model to obtain object description information and object-related text information corresponding to each key video frame; and a generation module for generating index information for the detection video based on a multimodal information set composed of multiple key video frames, object description information corresponding to each key video frame, and object-related text information, so as to determine the information set using the index information when receiving retrieval information for the detection video.
[0018] Another aspect of this disclosure provides a video retrieval apparatus, comprising: a receiving module for responding to receiving retrieval information from a user, wherein the retrieval information includes at least one of: first sub-retrieval information in an image modality and second sub-retrieval information in a text modality; and a retrieval module for determining target index information matching the retrieval information from a plurality of candidate index information to obtain a target detection video indicated by the target index information, wherein the candidate index information is generated according to the above-described index information generation method.
[0019] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the methods described above.
[0020] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.
[0021] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.
[0022] In the embodiments of this disclosure, dynamic sampling of the detection video based on its content and further filtering based on object information can reduce the workload of subsequent multimodal large-scale models while obtaining clear and effective key video frames, ensuring the accuracy of text recognition and extraction. Furthermore, after extracting object description information and object-related text information, the key video frames from which the text is extracted, along with the object description information of the extracted objects, can be combined into an information set and index information generated. This facilitates rapid video frame localization based on the index information, resulting in a better user experience. Attached Figure Description
[0023] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:
[0024] Figure 1 An exemplary system architecture for applying index information generation methods and video retrieval methods according to embodiments of this disclosure is illustrated.
[0025] Figure 2 A flowchart illustrating an index information generation method according to an embodiment of the present disclosure is shown.
[0026] Figure 3 The diagram illustrates a data flow graph of generating and storing information using a multimodal large model, according to an embodiment of the present disclosure.
[0027] Figure 4 The diagram illustrates the data flow of key video frames selected according to an embodiment of the present disclosure.
[0028] Figure 5 The flowchart illustrating the process of obtaining key video frames through clustering screening according to an embodiment of the present disclosure is shown in the illustration.
[0029] Figure 6 The flowchart illustrating the process of obtaining key video frames by filtering motion change information according to an embodiment of the present disclosure is shown.
[0030] Figure 7 A flowchart illustrating a video retrieval method according to an embodiment of the present disclosure is shown schematically.
[0031] Figure 8 A block diagram of an index information generation apparatus according to an embodiment of the present disclosure is shown schematically.
[0032] Figure 9 A block diagram of a video retrieval apparatus according to an embodiment of the present disclosure is shown schematically.
[0033] Figure 10 A block diagram schematically illustrates an electronic device suitable for implementing an index information generation method and a video retrieval method according to embodiments of the present disclosure. Detailed Implementation
[0034] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.
[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0036] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0037] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).
[0038] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.
[0039] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.
[0040] In security inspection scenarios, the objects being inspected are typically moving, moving slowly, and may be obscured by various buildings. For example, when inspecting vehicles, license plates are easily obscured by people or buildings, and the driver's body may be obscured by the vehicle itself. Throughout the inspection process, the driver needs to move the vehicle slowly for inspection. Furthermore, security inspection scenarios may involve various vehicles from multiple regions, each with different outlines and brand shapes. OCR is only suitable for standard printed text and has a low recognition rate for complex layouts, handwritten text, or low-resolution images, resulting in a low text recognition rate in security inspection scenarios. Moreover, without adjusting parameters for the inspected object, the text recognition rate of OCR is even lower. Additionally, while deep learning algorithms can perform text recognition and extraction from inspection videos, they require significant computational resources for the video dimension and can only recognize and extract text from training targets.
[0041] Furthermore, both of these methods are highly sensitive to image quality; blurry, skewed, or shadowed images significantly reduce recognition rates. In unforeseen circumstances requiring further identification and extraction of new information within the detected video, the two methods suffer from low extraction efficiency for this new information due to the lack of trained extraction parameters, and are ill-suited to handling unexpected situations. In security detection scenarios requiring backtracking of detected videos, these methods struggle to locate video frames containing text, negatively impacting the user experience.
[0042] In summary, the embodiments of this disclosure provide an index information generation method that can not only accurately extract text from objects within a detected video, but also combine key video frames from the extracted text with object description information of the extracted objects into an information set, which facilitates subsequent video frame localization and provides a better user experience.
[0043] Figure 1 This illustration schematically depicts an exemplary system architecture to which the index information generation method and video retrieval method can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.
[0044] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, an image acquisition unit 104, a server 105, and a network 106. The network 106 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.
[0045] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 106 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).
[0046] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0047] The image acquisition unit 104 is used to acquire detection video.
[0048] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.
[0049] It should be noted that the index information generation method provided in this embodiment can generally be executed by server 105. Correspondingly, the index information generation device provided in this embodiment can generally be located in server 105. The index information generation method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the index information generation device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the index information generation method provided in this embodiment can also be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the index information generation device provided in this embodiment may also be disposed in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or disposed in other terminal devices different from the first terminal device 101, the second terminal device 102, and the third terminal device 103.
[0050] For example, the image acquisition unit 104 can send the acquired detection video to the server 105, whereby the server 105 executes the aforementioned index information generation method. Users can send retrieval information through any one of the first terminal device 101, the second terminal device 102, or the third terminal device 103 to determine the information set based on the retrieval information and the index information.
[0051] It should be understood that Figure 1 The number of terminal devices, networks, image acquisition units, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0052] Figure 2 A flowchart illustrating an index information generation method according to an embodiment of this disclosure is shown schematically. Figure 2 As shown, the method includes operations S210~S240.
[0053] In operation S210, the detection video is dynamically sampled according to the video content of the detection video to obtain multiple sampled video frames, wherein the sampled video frames include the target object.
[0054] Inspection videos can be videos designed for security inspection scenarios, with the target object being the object being inspected. For example, an inspection video might be used to inspect vehicles, people, or ships, with the target object being the aforementioned vehicles, people, or ships. Multiple video frames in the inspection video include the target object to be inspected. The target object changes dynamically within the inspection video, but the rate of change is usually slow. For inspection videos containing a large amount of information, dynamic selection can be made based on the video content to reduce the amount of data required for subsequent analysis, while ensuring that the video frames analyzed later achieve the desired purpose.
[0055] The video content to be detected can include whether the target object appears in the video, or whether the video is clear. Therefore, for different video segments within the detection video, a sampling rate can be determined based on the detection content of each segment, and sampling can be performed using this sampling rate to obtain multiple sampled video frames. It is understandable that, to ensure subsequent analysis of the target object, the sampled video frames obtained when sampling based on the video content include the target object.
[0056] For example, the detection video can be a 3-minute video. If the target object appears in a 1-2 minute segment, the 1-2 minute segment can be sampled to obtain multiple sampled video frames that include the target object.
[0057] In operation S220, multiple key video frames are selected from multiple sampled video frames based on object information of the target object in multiple sampled video frames.
[0058] Object information for a target object is used to represent the characteristics of the target object. The target object can contain various types of information, such as its location, shape, and outline.
[0059] Because the target object changes slowly in the detected video, the similarity or repetition among multiple sampled video frames is high. Therefore, filtering can be performed based on object information related to the target object within multiple sampled video frames. Understandably, the number of key video frames obtained through filtering is less than the number of sampled video frames.
[0060] A key video frame can be a video frame that clearly shows the target object, so that text can be identified and extracted from the target object based on the key video frame, such as extracting license plate numbers from vehicles or recognizing car logos.
[0061] When operating S230, a multimodal large model is used to analyze multiple key video frames to obtain object description information and object-related text information corresponding to each key video frame.
[0062] Multimodal large language models (MLLMs) can perform multimodal tasks involving multiple types of information, such as text, images, and audio. For example, MLLMs can extract text from videos and describe videos semantically.
[0063] Object description information can be text used to describe the target object. Object description information includes text describing the characteristics of the target object and text describing the scene in which the target object is located. For example, if the target object is a vehicle, in a security inspection scenario, the object description information could be: "A vehicle of model XX is slowly moving towards the security checkpoint, and the driver is wearing a blue jacket." Here, "A vehicle of model XX is slowly moving towards the security checkpoint" can be text describing the scene in which the target object is located, while "A vehicle of model XX and the driver are wearing a blue jacket" are text describing the characteristics of the target object.
[0064] Object-related text information refers to text extracted from a target object that can identify the target object. For example, object-related text information could be a vehicle's license plate number or vehicle logo.
[0065] After filtering key video frames from the detected video through multiple screenings, object description information and object-related text information can be extracted from each key video frame using a multimodal large model.
[0066] In the embodiments of this disclosure, in addition to using a multimodal large model to identify and extract structured object-related text information, structured object description information is also extracted and generated from unstructured video frames, which facilitates subsequent retrieval based on structured information.
[0067] In operation S240, an index information for the detected video is generated based on a multimodal information set consisting of multiple key video frames, object description information corresponding to each key video frame, and object-related text information. When retrieval information for the detected video is received, the information set is determined based on the retrieval information and the index information.
[0068] Since the object description information and object-related text information are extracted from key video frames, and each key video frame in the detection video has a unique timestamp, the key video frames, their corresponding object description information, and object-related text information can be stored together. This allows the key video frames of the video modality / image modality to be uniquely associated with the object description information and object-related text information of the text modality via timestamps. Similarly, after storing multiple key video frames of the detection video together via timestamps, the above information can be combined into an information set for the detection video, and corresponding index information can be generated based on the information set.
[0069] For example, the index information may include: the identified target object, the name of the detected video, and the extracted object-related text information. Furthermore, to locate video frames and object-related text information within the information set based on the index information, the index information may also include object descriptions of multiple key video frames.
[0070] During the video review phase, if a user needs to search for the detected video, they can enter search information. The backend will then locate the relevant information set by comparing the search information with the index information. Furthermore, if a specific key video frame needs to be located, it can be determined from the information set based on more detailed information from the search results. For example, the search information could be: "A vehicle with license plate number BB arrived at the detection checkpoint." By matching "license plate number BB" with the index information, the corresponding information set can be retrieved. Further, by comparing the semantics of the search information with the semantics of the object description information within the information set, the corresponding key video frame can be located.
[0071] In the embodiments of this disclosure, dynamic sampling of the detection video based on its content and further filtering based on object information can reduce the workload of subsequent multimodal large-scale models while obtaining clear and effective key video frames, ensuring the accuracy of text recognition and extraction. Furthermore, after extracting object description information and object-related text information, the key video frames from which the text is extracted, along with the object description information of the extracted objects, can be combined into an information set and index information generated. This facilitates rapid video frame localization based on the index information, resulting in a better user experience.
[0072] According to embodiments of this disclosure, for operation S240, a multimodal large model is used to analyze multiple key video frames respectively to obtain object description information and object-related text information corresponding to each key video frame. This includes: for each key video frame, using a multimodal large model to perform semantic analysis on the target object in the key video frame to obtain object description information; and using a multimodal large model to perform text recognition on the target object to obtain object-related text information.
[0073] Key video frames and prompts can be input into a multimodal large model to guide the model to perform semantic analysis and text recognition on the target object, thereby obtaining object description information and object-related text information.
[0074] Cue words can include those used to guide multimodal large models in performing semantic analysis and text recognition tasks.
[0075] For example, the prompts may include: "Please extract the object-related text information from the target objects in each of the following key video frames, and describe the target objects in each key video frame, outputting the information in text form." The target objects may be vehicles, people, drivers, etc. "Please extract the object-related text information from the target objects in each of the following key video frames" is a prompt used to guide the multimodal large model to perform text recognition tasks, while "Describe the target objects in each key video frame and output the information in text form" is a prompt used to guide the multimodal large model to perform semantic analysis tasks. In this embodiment, the prompts also provide object hints commonly found in the current scene. Through object hints, the multimodal large model can understand which objects in the key video frames need to be analyzed specifically, without needing to train the multimodal large model.
[0076] In one embodiment, key video frames can be enhanced at low resolution before being input into a multimodal large model.
[0077] In the embodiments of this disclosure, the use of a multimodal large model to process key video frames can adapt to various types of target objects in security detection scenarios and accurately extract object-related text information without requiring model training for different types. Furthermore, the multimodal large model can generate object description information by understanding the internal semantics and logic of key video frames, facilitating subsequent generation of index information and video frame backtracking without increasing training and generation costs.
[0078] Figure 3 This schematically illustrates a data flow diagram of information generated and associatedly stored using a multimodal large model, according to an embodiment of this disclosure. For example... Figure 3 As shown, the detected video is dynamically sampled to obtain sampled video frames, and then further filtered using object information to obtain key video frames.
[0079] To ensure the extraction quality of the multimodal large model, key video frames can be enhanced at low resolution. These enhanced key video frames are then input into the multimodal large model, outputting object description information and object-related text information. Subsequently, the object description information, object-related text information, and key video frames can be stored as a multimodal information set.
[0080] In embodiments of this disclosure, information for different modalities may be stored in different physical spaces, but can be obtained through the same index information for the detected video.
[0081] According to an embodiment of this disclosure, for operation S220, multiple key video frames are selected from multiple sampled video frames based on object information of the target object in multiple sampled video frames, including: performing edge detection on the target object region of each sampled video frame to obtain edge detection results; selecting multiple intermediate video frames from multiple sampled video frames based on the edge detection results, wherein the multiple intermediate video frames are arranged in chronological order; and selecting multiple key video frames from multiple intermediate video frames based on object information of the target object in multiple intermediate video frames.
[0082] The target object region refers to the area that includes the target object. The size of the target object region can be determined based on the size of the target object or predetermined. In security detection scenarios, since the positions of detection equipment such as checkpoints are fixed, a fixed detection area can be determined based on the location of the detection equipment. This detection area is then used to detect the target object using a target recognition algorithm. After locating the target object, the area surrounding the target object is then defined as the target object region. Alternatively, the fixed detection area can be directly defined as the target object region.
[0083] Edge detection algorithms can be used to detect edges in the target object region. Examples of edge detection algorithms include the Harris Corner Detector, Canny Edge Detector, the Sobel operator, and Features from Accelerated Segment Test (FAST). The edge detection results can include the coordinates of the edge points extracted from the target object region.
[0084] In one specific embodiment, in order to save computing power and ensure the accuracy of text extraction, the relative position between the text region and the target object can be determined based on prior knowledge. Then, edge detection is performed only on the text region in the target object region.
[0085] After obtaining the edge detection results, continuity analysis can be performed to filter out intermediate video frames with continuous edges from multiple sampled video frames. Continuity analysis can be performed using methods such as connectivity analysis, contour closure analysis, and edge continuity detection to obtain continuity analysis values. If the continuity analysis value of a sampled video frame is greater than or equal to a continuity threshold, that sampled video frame is considered an intermediate video frame. For example, if the integrity of the contour edges of the target object region or the proportion of characters is ≥85% (such as continuous and unbroken edges of license plate characters or clear edges of facial features), an intermediate video frame is determined to eliminate blurry frames with fragmented edges.
[0086] After selecting multiple intermediate video frames, they can be arranged in chronological order. Then, based on the object information of the target object in the multiple intermediate video frames, the similarity and motion differences of the target object can be determined, so as to select multiple key video frames from the multiple intermediate video frames.
[0087] In the embodiments of this disclosure, edge detection is first performed on the target object region to obtain intermediate video frames with clear edges. Further, filtering is performed based on the object information of the target object, which helps improve the accuracy of object-related text information. Moreover, since detection is performed only on a localized target object region, computational power consumption is low.
[0088] According to embodiments of this disclosure, the object information includes: a spliced vector obtained by concatenating edge feature vectors and grayscale statistical vectors, wherein the edge feature vectors include the coordinates of multiple edge points obtained after edge detection of the target object region, and the grayscale statistical vectors include multiple grayscale index values for the target object region; based on the object information for the target object in multiple intermediate video frames, multiple key video frames are selected from the multiple intermediate video frames, including: dividing the multiple intermediate video frames into multiple first video segments according to time order; and selecting key video frames from each first video segment based on the spliced vectors of the multiple intermediate video frames in each first video segment.
[0089] For example, taking the Canny edge detection algorithm as an example, the edge detection result can be the coordinates of the edge points of the character outline. For example, for the edges of the 7 characters of a license plate number, 20 key edge points are taken for each character to form an edge feature vector.
[0090] Grayscale index values can be used to characterize the brightness or darkness of a target object region. It should be noted that in security detection scenarios, the security detection time for the target object is relatively short, and the scene lighting changes slowly within the detection video, resulting in stable grayscale features. Therefore, grayscale index values can be used to assist in selecting clear frames to avoid grayscale fluctuations caused by lighting / noise, thus preventing them from affecting the accuracy of subsequent text recognition and extraction.
[0091] For example, multiple grayscale index values can be the grayscale mean, grayscale variance, grayscale entropy, etc. of the target object area.
[0092] In this embodiment, to ensure screening accuracy and avoid the influence of video frames with reflections (such as those from car windows) or character fluctuations over a long period, multiple intermediate video frames can be divided into multiple first video segments according to time sequence. Each first video segment includes a predetermined number of intermediate video frames. For example, a first video segment may include 3 to 5 intermediate video frames.
[0093] For each first video segment, clustering is performed using the spliced vectors of multiple intermediate video frames in each first video segment, and a key video frame is selected from each first video segment based on the clustering results.
[0094] According to embodiments of this disclosure, in order to improve computational efficiency, clustering and other operations can be distributed to multi-threaded parallel processing.
[0095] In the embodiments of this disclosure, by concatenating the edge feature vector and the grayscale statistical vector, the concatenated vector can be obtained by utilizing the inter-frame edge consistency and illumination consistency, respectively. Then, the concatenated vector is used to filter out clear and illuminated key video frames from the intermediate video frames, thereby reducing the amount of data to be analyzed and further improving the accuracy of subsequent text recognition and extraction.
[0096] Figure 4 The diagram illustrates the data flow of key video frames selected according to an embodiment of this disclosure. For example... Figure 4 As shown, for the detection video, multiple sampled video frames are dynamically sampled based on the video content. Further, edge detection is performed on the target object region within the sampled video frames, and multiple intermediate video frames are obtained based on the edge detection results. Further, clustering can be performed using the concatenated vectors corresponding to the intermediate video frames to obtain the final key video frames used in the multimodal large model. Through these multiple filtering steps, clear and less repetitive key video frames can be gradually selected for security detection scenarios, thereby further improving the accuracy of subsequent text recognition and extraction.
[0097] The following will be through Figure 5 Further explain the specific process of clustering.
[0098] Figure 5 The flowchart illustrating the process of obtaining key video frames through clustering screening according to an embodiment of this disclosure is shown. Figure 5 As shown, the method includes operations S510~S540. For each first video segment,
[0099] In operation S510, the cluster center of the first video segment is determined based on the valid information of the first intermediate video frame in the first video segment.
[0100] When operating the S520, clustering is performed on the spliced vectors of multiple intermediate video frames based on the cluster centers to obtain the clustering results.
[0101] In operation S530, based on the clustering results, the target splicing vector that meets the clustering conditions is determined from multiple splicing vectors.
[0102] In operation S540, the intermediate video frame corresponding to the target splicing vector is used as the key video frame for the first video segment.
[0103] Before clustering, to avoid randomized cluster centers affecting clustering accuracy, the cluster centers of the first video segment can be determined based on the valid information of the first intermediate video frame in the first video segment.
[0104] In one embodiment, the valid information of the first intermediate video frame may include edge integrity information determined based on the coordinates of edge points (such as the contour edge integrity mentioned above) and grayscale information determined based on grayscale index values (such as the grayscale variance mentioned above). For example, if the contour edge integrity is ≥85% and the grayscale variance is ≥80, the first intermediate video frame is determined to be a valid frame. Alternatively, the valid information may also include noise information of the target object region, such as noise density. For example, if the contour edge integrity is ≥85%, the grayscale variance is ≥80, and the noise density is less than or equal to a noise threshold (such as 12%), the first intermediate video frame is determined to be a valid frame. When the first intermediate video frame is a valid frame, the stitched vector of the first intermediate video frame is used as the cluster center.
[0105] In another embodiment, the effective information may include only noise information, and the noise information may be quantified at a more granular level. For example, if the noise density of the first intermediate video frame is less than or equal to the noise threshold and greater than or equal to the boundary noise threshold, such as when the noise density is between 8% and 12%, the first intermediate video frame is determined to have slight noise. In this case, intermediate video frames with noise densities all less than or equal to the noise threshold can be identified from the first video segment, and the mean of the splicing vectors of these intermediate video frames can be used as the cluster center. For example, if the first video segment includes three intermediate video frames, t0, t1, and t2, and the first intermediate video frame (t0) has slight noise (noise density 8%-12%), then the mean of the splicing vectors of t0 and t2 (F0+F2) / 2 can be used as the cluster center (C0) to match the characteristic stability of the security detection short frame sequence (t0~t2).
[0106] After determining the cluster centers, for each newly added splicing vector, the distance between the splicing vector and the cluster center (such as Euclidean distance) is determined. If the distance is greater than the clustering threshold, the cluster center is updated to avoid frequent fluctuations in the cluster centers caused by noisy frames.
[0107] For example, still using t0, t1, and t2 as examples, if the Euclidean distance between the newly added t1 and the cluster center is greater than 8 (determined according to the security detection scenario), the cluster center is updated using a weighted average. The updated cluster center is: C_new = (C × (n-1) + F_new) / n, where n is the number of the first video frames, C_new and C are the updated cluster center and the original cluster center, respectively, and F_new is the concatenated vector of the newly added intermediate video frames. It should be noted that, since feature changes are slow in the security detection scenario, a full update is not required during clustering; only the cluster centers are updated.
[0108] The clustering result is the cluster centers when the clustering stopping condition is met. In one embodiment, the clustering stopping condition can be that the number of updates reaches a threshold, such as 40% of the total number of intermediate video frames in the first video segment (e.g., 5 × 40% = 2), to prevent feature shift due to excessive iteration. In another embodiment, the clustering stopping condition can be that the clustering error reaches an error threshold, where the clustering error refers to the average distance between the clustered video frames and the cluster centers (e.g., the error threshold could be 5). Since feature consistency is high and error tolerance is low in security detection scenarios, an error threshold can be used for limitation. In yet another embodiment, the clustering stopping condition can also be that the offset of the target object's position in an intermediate video frame from a fixed region is greater than a threshold. For example, the fixed region can be the area around the detection device to delete video frames that are about to leave the inspection area, retaining the current valid frames and avoiding information loss. This fixed region can be called the Region of Interest (ROI).
[0109] For a given clustering result (i.e., cluster centers), the target stitching vector that satisfies the clustering conditions is determined from the stitching vectors closest to the cluster centers. If multiple target stitching vectors exist, the target stitching vector with lower noise density and higher grayscale variance (clearer) is selected as the final target stitching vector to filter out a unique key video frame from the first video segment.
[0110] In the embodiments of this disclosure, by determining cluster centers and clustering the spliced vectors, clear and well-lit key video frames can be obtained by using inter-frame edge consistency and illumination consistency clustering. This reduces the amount of data to be analyzed while further improving the accuracy of subsequent text recognition and extraction.
[0111] According to embodiments of this disclosure, the noise information is obtained after noise detection of the target object region. The noise information of the target object region may include noise density. For example, for targeted detection of salt-and-pepper noise, the proportion of pixel grayscale abrupt change points within the target object region can be used as the noise density. In security detection scenarios, noise information is typically caused by vehicle glass reflections, temporary lighting interference, and partial obstruction during document handover.
[0112] According to embodiments of this disclosure, the object information includes noise information of the target object region; multiple intermediate video frames are arranged in chronological order; multiple key video frames are obtained by filtering from the multiple intermediate video frames based on the object information of the target object in the multiple intermediate video frames, including: deleting intermediate video frames with noise information greater than a noise threshold from the multiple intermediate video frames based on the noise information of the target object region in each intermediate video frame, to obtain updated multiple intermediate video frames; and obtaining multiple key video frames by filtering from the updated multiple intermediate video frames.
[0113] Noisy video frames may be interspersed among multiple intermediate video frames arranged in chronological order. Before selecting key video frames, intermediate video frames with high noise levels should be filtered out. For example, intermediate video frames with a noise density greater than 12% can be identified from multiple intermediate video frames and then filtered out.
[0114] In one embodiment, after filtering out noisy intermediate video frames, key video frames can be selected based on the similarity between the updated intermediate video frames. For example, key video frames can be selected based on structural similarity index measure (SSIM).
[0115] For example, for multiple updated intermediate video frames, the SSIM between two adjacent intermediate video frames can be determined. If the SSIM of M consecutive intermediate frames is greater than or equal to the structural similarity threshold (e.g., 0.93), a key video frame can be selected from the M+1 intermediate video frames corresponding to the M SSIMs, where M is a positive integer greater than 2.
[0116] Taking intermediate video frames t0, t1, and t2 as an example, if the noise density of t1 is greater than 12% or the grayscale variance is greater than 18, t1 is discarded, resulting in updated intermediate video frames t0 and t2. If the SSIM between t0 and t2 is greater than 0.93, a key video frame is selected from t0 and t2, such as the one with lower noise density or higher grayscale variance; otherwise, both t0 and t2 are key video frames.
[0117] In another embodiment, the noise removal process of the intermediate video frames can be used as a preprocessing step for clustering. The updated intermediate video frames are divided into multiple first video segments according to the time sequence, and key video frames are filtered. The specific operation is described above and will not be repeated here.
[0118] In the embodiments of this disclosure, noise information targeting the target object region is used to filter out noise frames from multiple intermediate video frames that are consecutive in time, so as to avoid the noise information affecting the filtering accuracy of key video frames.
[0119] It is understood that the above method utilizes changes in text edges and grayscale to filter key video frames. Furthermore, embodiments of this disclosure also support analysis based on motion changes of the target object to filter key video frames; specific operations are described below.
[0120] Figure 6 The flowchart illustrating the process of obtaining key video frames through motion change information according to an embodiment of this disclosure is shown. Figure 6 As shown, the method includes operations S610~S620.
[0121] According to embodiments of this disclosure, the object information includes location information of the target object region; and object information for the target object is obtained from multiple intermediate video frames. The following describes how, for each first video segment, multiple key video frames are obtained by filtering from the multiple intermediate video frames through operations S610-S620.
[0122] In operation S610, the motion change information of the target object between multiple intermediate video frames is determined based on the position information of the target object region in each intermediate video frame.
[0123] When operating the S620, multiple key video frames are selected from multiple intermediate video frames based on motion change information.
[0124] The location information of the target object region can be the center coordinates of the target object region, the coordinates of the edge points of the text region within the target object region, or the center coordinates of the text region. The text region can be determined based on prior knowledge, as mentioned above, and the coordinates of the edge points can be obtained through edge detection as described above.
[0125] The motion change information of the target object between multiple intermediate video frames can include: the positional information of the target object region relative to the static background and the distance it has moved in two adjacent intermediate video frames.
[0126] For example, after determining motion change information (the distance the target object moves between two adjacent intermediate video frames), a key video frame can be selected from multiple intermediate video frames with relatively small movement distances.
[0127] In the embodiments of this disclosure, by using the motion change information of the target object between multiple intermediate video frames, key video frames with positional differences can be quickly identified, reducing the computational power of screening and improving screening efficiency.
[0128] According to a specific embodiment of this disclosure, multiple key video frames are selected from multiple intermediate video frames based on motion change information, including: if the motion change information of the target object between two adjacent intermediate video frames satisfies the motion difference condition, a key video frame is determined from the two adjacent intermediate video frames.
[0129] For example, if the movement distance of the target object between two adjacent intermediate video frames is less than or equal to a distance threshold (e.g., 3 pixels), the key video frame with the larger grayscale variance is determined from the two adjacent intermediate video frames. The motion difference condition is: the movement distance between two adjacent intermediate video frames is less than or equal to the distance threshold. The movement distance can be Euclidean distance. If the movement distance of the target object between two adjacent intermediate video frames is greater than the distance threshold, then both adjacent intermediate video frames are considered key video frames.
[0130] In the embodiments of this disclosure, by analyzing the motion change information between two adjacent intermediate video frames, it is possible to achieve inter-frame motion difference analysis and quickly filter out key video frames.
[0131] Furthermore, in security detection scenarios, people and vehicles move relatively slowly. While adjacent frames may show high similarity across multiple intermediate video frames, the differences between people and vehicles become significant when accumulated across multiple frames. Therefore, to select a suitable number of key video frames from multiple intermediate frames that clearly demonstrate positional differences, the method of accumulating movement distance can also be used.
[0132] According to another specific embodiment of this disclosure, multiple key video frames are selected from multiple intermediate video frames based on motion change information, including: taking the intermediate video frame that satisfies a first condition as the first frame, determining multiple intermediate video frames adjacent to the intermediate video frame that satisfies the first condition from the multiple intermediate video frames to obtain a second video segment; determining cumulative motion change information based on the motion change information of the target object in the second video segment; and determining at least one key video frame from the second video segment using a selection strategy that matches the cumulative motion change information.
[0133] Understandably, when selecting key video frames, if the target object or the text to be extracted is occluded in the first intermediate video frame, then that frame is meaningless for subsequent multimodal large model analysis. Therefore, the first condition can be that the integrity of the contour edges of the intermediate video frames meets the corresponding conditions. The conditions for contour edge integrity can be found above and will not be repeated here.
[0134] After selecting an intermediate video frame that meets the first condition from multiple intermediate video frames, and using that frame as the first frame, select multiple intermediate video frames that are adjacent to and follow that frame in chronological order to form the second video segment. For example, if frame t0 meets the first condition, then 6 intermediate video frames, t0, t1, t2, t3, t4, and t5, can be selected as the second video segment.
[0135] For the selected second video segment, the sum of the movement distances of the target object region between two adjacent intermediate video frames in the second video segment can be used as the cumulative motion change information. Alternatively, the movement distance between the first and last intermediate video frames in the second video segment can be directly used as the cumulative motion change information.
[0136] Based on the relationship between cumulative motion change information and a cumulative distance threshold, a matching filtering strategy is determined to identify at least one key video frame from the second video segment. For example, if the movement distance exceeds a certain threshold, a specific number of intermediate video frames can be selected as key video frames from specific locations within the second video segment.
[0137] For example, consider a second video segment comprising t0~t5, where the cumulative motion change information includes a first sub-movement distance determined based on the center coordinates of the text region or the target object region, and a second sub-movement distance determined based on the coordinates of the edge points of the text region. The first and second sub-movement distances are used to characterize the global cumulative offset of the target object region and the cumulative offset of the edge points, respectively.
[0138] In the embodiments of this disclosure, by performing cumulative operation change statistics on a second video segment composed of multiple consecutive intermediate video frames, it is possible to simultaneously filter out key video frames with clear edges and a small number of frames, thereby ensuring the accuracy and efficiency of subsequent text extraction.
[0139] For scenario A, if the first sub-movement distance is less than or equal to the first cumulative threshold (e.g., 8 pixels) and the second sub-movement distance is less than or equal to the second cumulative threshold (e.g., 15 pixels), the middle video frame with the largest grayscale variance in the second video segment can be used as the key video frame, such as t3. Scenario A is typically a security detection scenario where the target object is completely stationary (e.g., driver stop verification). In this case, only one frame needs to be retained, instead of retaining multiple frames.
[0140] For scenario B, where the first sub-movement distance is greater than the first cumulative threshold and the second sub-movement distance is less than or equal to the second cumulative threshold, the middle video frames located at the beginning, end, and middle positions of the second video segment can be used as key video frames (e.g., t0, t3, and t5, a total of 3) to filter out duplicate video frames. Scenario B is typically a security detection scenario where the target object moves slowly (e.g., moving forward more than 8 pixels along the lane). In this case, it is necessary to cover the difference in the beginning and end positions to avoid missing the license plate edge in subsequent detections.
[0141] For scenario C, where the first sub-movement distance is less than or equal to the first cumulative threshold and the second sub-movement distance is greater than the second cumulative threshold, the two intermediate video frames with the largest distance between edge points in the second video segment can be selected as key video frames (e.g., if the distance between the same edge point is largest in t2 and t5, then t2 and t5 are selected) to filter out video frames with no edge variation. Scenario C is typically a scenario in security detection where lighting changes slowly (e.g., cloud cover causing changes in the brightness of license plate edges). In this case, the frame with the clearest edges should be retained.
[0142] For scenario D, where the first sub-movement distance exceeds the first cumulative threshold and the second sub-movement distance exceeds the second cumulative threshold, multiple intermediate video frames can be selected from the second video segment at predetermined intervals as key video frames to ensure coverage of both the initial and final positions and clear edges. Scenario D typically occurs in security detection scenarios where the target object moves slowly with slight occlusion (such as a passenger reaching out to hand over their ID).
[0143] According to an embodiment of this disclosure, for operation S210, dynamic sampling of the detection video is performed based on the video content of the detection video to obtain multiple sampled video frames, including: performing object recognition on a fixed region in the detection video to obtain a first detection result, wherein the fixed region corresponds to the detection location targeted by the detection video; performing quality detection and illumination detection on the target object region of the detection video to obtain a second detection result and a third detection result, wherein the target object region is a region including the target object; performing inter-frame structural similarity detection on the detection video to obtain a fourth detection result; determining multiple sampling frequencies for the detection video based on at least one of the first detection result, the second detection result, the third detection result, and the fourth detection result; and sampling the detection video using the multiple sampling frequencies to obtain multiple sampled video frames.
[0144] A fixed area can be the area surrounding a detection device used for security checks, also known as the Region of Interest (ROI) as mentioned above, such as a lane or verification window. For the ROI, object recognition algorithms can be used for object identification, such as a lightweight model based on YOLO (You Only Look Once). The first detection result can be used to characterize whether a target object was detected in the fixed area; if a target object was detected, it can also include whether the target object completely falls within the ROI.
[0145] Quality detection and illumination detection are performed on the quality of video frames and the illumination, respectively, and can be achieved through corresponding detection metrics. Correspondingly, the second and third detection results can be detection metric values. For example, quality detection metrics may include: grayscale variance (≥80 indicates sharpness, <80 indicates blurriness), edge strength (calculated based on the Sobel operator, ≥0.6 indicates intact edges), and occlusion degree (≤10% indicates no effective occlusion, >10% indicates invalid occlusion), etc. In addition, quality detection can also detect the stability of the target object's position. Quality detection metrics may also include: the offset of the target object region (because the position is fixed in security detection scenarios, an offset ≤±5 pixels indicates a stable position, and an offset exceeding this indicates an unstable position). Illumination detection metrics may include: average brightness fluctuation (≤±20 nits, to avoid OCR failure caused by backlighting / shadows).
[0146] Inter-frame structural similarity detection is the calculation of the structural similarity (SSIM) between two adjacent video frames in the detection video. The fourth detection result is the specific structural similarity value (SSIM ≥ 0.95 is considered highly redundant, < 0.95 is considered a valid change. This threshold is a global threshold for video frames and is usually greater than the SSIM threshold of 0.93 for the target object region).
[0147] In the embodiments of this disclosure, the sampling frequency can be determined based on any one of the four detection results mentioned above, so as to use the sampling frequency corresponding to the video content for dynamic sampling, which reduces the amount of data and ensures the quality foundation of subsequent keyframe screening.
[0148] According to embodiments of this disclosure, determining multiple sampling frequencies for a detection video based on at least one of a first detection result, a second detection result, a third detection result, and a fourth detection result includes: determining a first sampling frequency when a third video segment in the detection video meets a second condition, wherein the second condition includes: the first detection result indicates that the target object is located within a fixed area, the second detection result indicates that the target object area meets a quality condition, and the fourth detection result indicates that the inter-frame structural similarity of the third video segment is greater than a first threshold; determining a second sampling frequency when a fourth video segment in the detection video meets a third condition, wherein the third condition includes one of the following: the first detection result indicates that a portion of the target object is located within a fixed area, the second detection result indicates that the target object area meets a quality condition, and the fourth detection result indicates that the inter-frame structural similarity of the fourth video segment is greater than a second threshold and less than a first threshold; determining a third sampling frequency when a fifth video segment in the detection video meets a fourth condition, wherein the third condition includes one of the following: the second detection result indicates that the target object area does not meet a quality condition, and the third detection result indicates that the target object area does not meet an illumination condition; the frequency values of the first sampling frequency, the second sampling frequency, and the third sampling frequency increase sequentially.
[0149] For example, the first sampling frequency, the second sampling frequency, and the third sampling frequency are low (e.g., 1-2 frames / second), medium (e.g., 4-6 frames / second), and high sampling rates (e.g., 8-10 frames / second), respectively.
[0150] The detection video may include at least one of a second video segment, a third video segment, and a fourth video segment. For ease of understanding, the following three embodiments will be used as examples.
[0151] In one embodiment, the second condition for the third video segment is: the target object is present within the ROI + the position is stable (offset ≤ ±5 pixels) + it is clear and unobstructed (grayscale variance ≥ 80 + occlusion ≤ 10%) + SSIM ≥ 0.95 (first threshold). Since the target object moves slowly in security detection scenarios, low-frequency frames can already cover effective information, eliminating the need for high-frequency sampling. To reduce computational power consumption, only clarity and positional stability are required, and video frames with high similarity are removed.
[0152] The third condition for the fourth video segment is: the target object has just entered the ROI (partially falling within the ROI) / slight positional fluctuation (5 < offset ≤ 10 pixels) / critical sharpness (60 ≤ grayscale variance < 80) / SSIM = 0.90-0.95. For scenarios where the target object "enters / moves slightly" (such as a vehicle slowly entering the inspection area), a medium sampling rate is used to balance sampling density and computing power to avoid missing keyframes.
[0153] The fourth condition for the fifth video clip is: the target object is occluded (>10%) / insufficient sharpness (grayscale variance <60) / significant positional fluctuation (offset >10 pixels) / sudden change in lighting. The sampling frequency is increased only when information loss is possible, capturing instantaneous sharp frames through a high sampling rate, such as when reflections from vehicle windows cause a decrease in sharpness.
[0154] In one specific embodiment, the fifth video segment usually coexists with the third or fourth video segment. This is because the location is relatively fixed in the security detection scenario, the high sampling rate only lasts for 2-3 seconds, and then drops back to the medium / low sampling rate after the target object stabilizes.
[0155] In the embodiments of this disclosure, the validity of whether the target is in place, the quality of the target object region, and the structural similarity are used together for dynamic sampling, which can reduce redundant video frames and ensure that valid frames are sampled, thus balancing sampling quality and the number of sampled frames.
[0156] Figure 7 A flowchart illustrating an embodiment of the video retrieval method of this disclosure is shown schematically. For example... Figure 7 As shown, the method includes operations S710~S720.
[0157] In operation S710, in response to receiving retrieval information from the user, wherein the retrieval information includes at least one of the following: a first sub-retrieval information in the image modality and a second sub-retrieval information in the text modality.
[0158] In operation S720, target index information matching the retrieval information is determined from multiple candidate index information to obtain the target detection video indicated by the target index information. The candidate index information is generated according to the above-mentioned index information generation method.
[0159] For example, the first sub-retrieval information for an image modality could be: given an input video frame to be retrieved, perform semantic analysis on the objects within the input video frame using a multimodal large model to obtain object description information. Then, based on the object description information of the input video frame, search the candidate index information (including object description information) in the index library to find object description information that matches the object description information of the input video frame, i.e., obtain the target index information. After determining the target index information, not only can the corresponding information set and target detection video be retrieved using the target index information, but the key video frame that matches the object description information of the input video frame can also be directly located.
[0160] For example, the second sub-retrieval information of the text modality could be: a vehicle with license plate number BB has arrived at the detection checkpoint. By matching the above retrieval information with multiple index information in the database, the target index information can be obtained, and then the corresponding information set and target detection video can be retrieved using the target index information.
[0161] If the search information includes both of the above modalities, please refer to the above text for the search operations for each modality.
[0162] In the embodiments of this disclosure, by constructing index information for associating and storing information sets (multimodal), users can perform dual "video + text" retrieval based on multimodal retrieval information when searching for detected videos, and quickly retrieve the target detected video and the multimodal information corresponding to a certain timestamp within its information set.
[0163] In summary, the index information generation method and video retrieval method provided in this disclosure can, on the one hand, filter clear frames (e.g., frames without blur) and fuse brightness information from multiple frames to restore the true features of text, thereby improving text recognition and extraction accuracy and overcoming the problem that single-frame OCR is prone to recognition failure due to motion blur (e.g., moving license plates) and sudden changes in illumination (e.g., scene switching in a video). On the other hand, the embodiments of this disclosure can track moving text on moving target objects (e.g., license plates of moving vehicles, moving price tags of goods in live broadcasts) and extract text in real time, avoiding the tedious manual frame-by-frame screenshotting and improving the ability to capture dynamic content.
[0164] Furthermore, the embodiments disclosed herein can be combined with "real-time streaming" technology to achieve end-to-end automated processing of detected videos, reducing manual intervention costs and improving business efficiency. For example, for existing video files (such as film and television libraries and surveillance recordings), the system can automatically extract text in batches to generate indexes, subtitles, or structured data (such as timestamps + text content).
[0165] Figure 8 A block diagram of an index information generation apparatus according to an embodiment of the present disclosure is shown schematically. Figure 8 As shown, the index information generation device 800 includes:
[0166] The sampling module 810 is used to dynamically sample the detection video based on the video content of the detection video to obtain multiple sampled video frames, wherein the sampled video frames include the target object.
[0167] The filtering module 820 is used to filter multiple key video frames from multiple sampled video frames based on object information of the target object in multiple sampled video frames.
[0168] Analysis module 830 is used to analyze multiple key video frames using a multimodal large model to obtain object description information and object-related text information corresponding to each key video frame.
[0169] The generation module 840 is used to generate index information for the detection video based on a multimodal information set consisting of multiple key video frames, object description information corresponding to each key video frame, and object-related text information, so as to determine the information set by using the index information when receiving retrieval information for the detection video.
[0170] Figure 9 A block diagram of a video retrieval device according to an embodiment of the present disclosure is shown schematically. Figure 9 As shown, the video retrieval device 900 includes:
[0171] The receiving module 910 is configured to respond to receiving retrieval information from a user, wherein the retrieval information includes at least one of the following: a first sub-retrieval information in an image modality and a second sub-retrieval information in a text modality.
[0172] The retrieval module 920 is used to determine the target index information that matches the retrieval information from multiple candidate index information, so as to obtain the target detection video indicated by the target index information, wherein the candidate index information is generated according to the above-mentioned index information generation method.
[0173] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.
[0174] For example, any plurality of the sampling module 810, filtering module 820, analysis module 830, and generation module 840 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the sampling module 810, filtering module 820, analysis module 830, and generation module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the sampling module 810, the filtering module 820, the analysis module 830, and the generation module 840 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.
[0175] Any multiple of the receiving module 910 and the retrieval module 920 can be combined into one module / unit / subunit or combined with at least some functions of other modules. The implementation process is as described above and will not be repeated here.
[0176] It should be noted that the apparatus portion in the embodiments of this disclosure corresponds to the method portion in the embodiments of this disclosure. The description of the apparatus portion is specifically referred to in the method portion, and will not be repeated here.
[0177] Figure 10 A block diagram schematically illustrates an electronic device suitable for implementing an index information generation method and a video retrieval method according to embodiments of the present disclosure. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0178] like Figure 10As shown, an electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0179] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.
[0180] According to embodiments of this disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the input / output (I / O) interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.
[0181] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0182] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.
[0183] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0184] For example, according to embodiments of this disclosure, a computer-readable storage medium may include one or more memories other than the ROM 1002 and / or RAM 1003 described above.
[0185] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.
[0186] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0187] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0188] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0189] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.
[0190] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.
Claims
1. A method for generating index information, characterized in that, The method includes: The detection video is dynamically sampled based on its content to obtain multiple sampled video frames, wherein each sampled video frame includes the target object. Based on the object information for the target object in the multiple sampled video frames, multiple key video frames are obtained by filtering from the multiple sampled video frames; Using a multimodal large model, multiple key video frames are analyzed separately to obtain object description information and object-related text information corresponding to each key video frame; and Based on a multimodal information set consisting of multiple key video frames, object description information corresponding to each key video frame, and object-associated text information, an index information for the detected video is generated so that when retrieval information for the detected video is received, the information set can be determined based on the retrieval information and the index information.
2. The method according to claim 1, characterized in that, The method utilizes a multimodal large model to analyze multiple key video frames separately, obtaining object description information and object-related text information corresponding to each key video frame, including: For each of the key video frames, the multimodal large model is used to perform semantic analysis on the target objects in the key video frames to obtain object description information; and Using the multimodal large model, text recognition is performed on the target object to obtain object-related text information.
3. The method according to claim 1, characterized in that, The step of filtering multiple key video frames from multiple sampled video frames based on object information for the target object in multiple sampled video frames includes: Edge detection is performed on the target object region of each of the sampled video frames to obtain edge detection results; Based on the edge detection results, multiple intermediate video frames are selected from the multiple sampled video frames, wherein the multiple intermediate video frames are arranged in chronological order; Based on the object information of the target object in the multiple intermediate video frames, multiple key video frames are obtained by filtering from the multiple intermediate video frames.
4. The method according to claim 3, characterized in that, The object information includes: a spliced vector obtained by concatenating an edge feature vector and a gray-level statistical vector, wherein the edge feature vector includes the coordinates of multiple edge points obtained after edge detection of the target object region, and the gray-level statistical vector includes multiple gray-level index values for the target object region; The step of filtering multiple key video frames from multiple intermediate video frames based on object information related to the target object in multiple intermediate video frames includes: According to the time sequence, the multiple intermediate video frames are divided into multiple first video segments; and The key video frame is obtained by selecting from each of the first video segments based on the splicing vector of multiple intermediate video frames in each of the first video segments.
5. The method according to claim 4, characterized in that, The step of selecting the key video frame from each of the first video segments based on the splicing vector of multiple intermediate video frames in each of the first video segments includes: for each of the first video segments, Based on the valid information of the first intermediate video frame in the first video segment, determine the cluster center of the first video segment; Based on the cluster centers, the splicing vectors of each of the multiple intermediate video frames are clustered to obtain clustering results; Based on the clustering results, a target splicing vector that satisfies the clustering conditions is determined from a plurality of splicing vectors; The intermediate video frame corresponding to the target splicing vector is used as the key video frame for the first video segment.
6. The method according to any one of claims 3 to 5, characterized in that, The object information includes noise information of the target object region; the multiple intermediate video frames are arranged in chronological order; the step of filtering multiple key video frames from the multiple intermediate video frames based on the object information of the target object in the multiple intermediate video frames includes: Based on the noise information for the target object region in each of the intermediate video frames, intermediate video frames with noise information greater than a noise threshold are deleted from the plurality of intermediate video frames to obtain updated intermediate video frames; and Multiple key video frames are obtained by filtering from multiple updated intermediate video frames.
7. The method according to claim 3, characterized in that, The object information includes the location information of the target object region; the step of filtering multiple key video frames from the multiple intermediate video frames based on the object information of the target object in the multiple intermediate video frames includes: Based on the position information of the target object region in each of the intermediate video frames, determine the motion change information of the target object between the multiple intermediate video frames; and Based on the motion change information, multiple key video frames are obtained by filtering from multiple intermediate video frames.
8. The method according to claim 7, characterized in that, The step of selecting multiple key video frames from multiple intermediate video frames based on the motion change information includes: If the motion change information of the target object between two adjacent intermediate video frames satisfies the motion difference condition, a key video frame is determined from the two adjacent intermediate video frames.
9. The method according to claim 7, characterized in that, The step of selecting multiple key video frames from multiple intermediate video frames based on the motion change information includes: The intermediate video frame that satisfies the first condition is taken as the first frame. From the multiple intermediate video frames, multiple intermediate video frames adjacent to the intermediate video frame that satisfies the first condition are determined to obtain the second video segment. Based on the motion change information of the target object in the second video segment, the cumulative motion change information is determined; At least one key video frame is determined from the second video segment using a filtering strategy that matches the cumulative motion change information.
10. The method according to claim 1, characterized in that, The step of dynamically sampling the detection video based on its content to obtain multiple sampled video frames includes: Object recognition is performed on a fixed region in the detection video to obtain a first detection result, wherein the fixed region corresponds to the detection location targeted by the detection video; The target object region of the detection video is subjected to quality detection and illumination detection respectively to obtain a second detection result and a third detection result, wherein the target object region is the region including the target object; The detection video is subjected to inter-frame structural similarity detection to obtain a fourth detection result; Based on at least one of the first detection result, the second detection result, the third detection result, and the fourth detection result, a plurality of sampling frequencies for the detection video are determined; The detected video is sampled using multiple sampling frequencies to obtain multiple sampled video frames.
11. The method according to claim 10, characterized in that, Determining multiple sampling frequencies for the detected video based on at least one of the first detection result, the second detection result, the third detection result, and the fourth detection result includes: If the third video segment in the detected video meets the second condition, a first sampling frequency is determined, wherein the second condition includes: the first detection result indicates that the target object is located in the fixed area, the second detection result indicates that the target object area meets the quality condition, and the fourth detection result indicates that the inter-frame structural similarity of the third video segment is greater than a first threshold. If the fourth video segment in the detected video meets the third condition, a second sampling frequency is determined, wherein the third condition includes one of the following: the first detection result indicates that part of the target object is located within the fixed region; the second detection result indicates that the target object region meets the quality condition; and the fourth detection result indicates that the inter-frame structural similarity of the fourth video segment is greater than the second threshold and less than the first threshold. If the fifth video segment in the detected video meets the fourth condition, a third sampling frequency is determined, wherein the third condition includes one of the following: the second detection result indicates that the target object region does not meet the quality condition, and the third detection result indicates that the target object region does not meet the illumination condition; The frequency values of the first sampling frequency, the second sampling frequency, and the third sampling frequency increase sequentially.
12. A video retrieval method, characterized in that, The method includes: In response to receiving retrieval information from a user, wherein the retrieval information includes at least one of the following: a first sub-retrieval information in an image modality and a second sub-retrieval information in a text modality; From multiple candidate index information, target index information matching the retrieval information is determined to obtain the target detection video indicated by the target index information, wherein the candidate index information is generated by the method according to any one of claims 1 to 11.
13. An index information generation device, characterized in that, The device includes: The sampling module is used to dynamically sample the detection video based on the video content of the detection video to obtain multiple sampled video frames, wherein the sampled video frames include the target object; The filtering module is used to filter multiple key video frames from the multiple sampled video frames based on object information for the target object in the multiple sampled video frames; The analysis module is used to analyze multiple key video frames using a multimodal large model to obtain object description information and object-related text information corresponding to each key video frame; and The generation module is configured to generate index information for the detected video based on a multimodal information set consisting of multiple key video frames, object description information corresponding to each key video frame, and object-associated text information, so as to determine the information set based on the retrieval information and the index information when receiving retrieval information for the detected video.
14. A video retrieval device, characterized in that, The device includes: A receiving module is configured to respond to receiving retrieval information from a user, wherein the retrieval information includes at least one of the following: a first sub-retrieval information in an image modality and a second sub-retrieval information in a text modality; A retrieval module is configured to determine target index information that matches the retrieval information from multiple candidate index information to obtain the target detection video indicated by the target index information, wherein the candidate index information is generated by the method according to any one of claims 1 to 11.
15. An electronic device comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 12.
16. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 12.
17. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 12.
