Cross-modal power video positioning method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of video understanding, and particularly discloses a cross-modal power video positioning method and system, electronic equipment and a storage medium, and the method comprises the steps: sequentially carrying out the visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation of each sample power video, and obtaining at least one target pseudo-supervision pair corresponding to each sample power video; each target pseudo supervision pair comprises a corresponding sample power video, a pseudo query text and starting and ending time of the pseudo query text in the sample power video; based on the target pseudo supervision pair corresponding to each sample power video, training a preset cross-modal power video positioning model to obtain a trained cross-modal power video positioning model; and inputting the to-be-detected power video and the query text into the trained cross-modal power video positioning model to obtain a video positioning result. According to the method, stable and efficient zero-sample cross-modal power abnormal behavior positioning is realized, and the accuracy and efficiency of cross-modal power video positioning are improved.

Description

Background Art

[0002] With the booming urban economy, the scale and demand of power engineering construction are growing, which puts forward more stringent and urgent requirements for the perception and monitoring of safety hazards in power production and construction scenes. As a key means of perception and detection, video surveillance plays an increasingly prominent role in real-time decision-making and accurate early warning. It is necessary to be able to quickly locate specific actions or events in unedited video content based on natural language queries. Therefore, cross-modal video positioning technology has become particularly important.

[0003] Cross-modal video localization is a crucial and challenging task in the field of video understanding. Its core goal is to accurately locate specific video moments in non-trimmed videos based on corresponding language queries. For example, given text queries such as "improper wearing of seat belts" and "improper wearing of labor protection supplies", this technology can accurately identify and locate the specific time intervals of these illegal operations in the video. However, most of the existing natural language video action localization technologies rely on full supervision or weak supervision conditions for large-scale model training, which requires manual annotation of the precise time constraint intervals of each action category, consuming a lot of manual annotation costs and making it difficult to adapt to complex and changeable actual situations, diverse illegal operations, and the needs of massive video surveillance. In contrast, zero-shot natural language video localization technology does not require any manual annotation, and can directly and automatically generate pseudo queries and corresponding video moment pairs to train the localization model, showing higher scalability and cost-effectiveness.

[0004] Research on natural language video action localization in zero-shot settings has made significant progress in the field of traditional video action localization and demonstrated excellent scalability to meet the growing needs of perceptual detection. However, research in this field still faces two major challenges:

[0005] ① Enhance the connectivity between segments and integrate contextual information: Current research methods often ignore the importance of context and treat visual segments as independent units. In video sequences, although the semantic context remains consistent, segmentation methods that rely solely on visual features will lead to fragmentation of visual segments. In addition, the isolation between segments causes the loss of contextual information, which not only limits the ability to infer the entire dynamic activity sequence from static image captions, but also affects the ability to establish temporal relationships between pseudo queries. To overcome this challenge, research needs to develop methods that can identify and integrate contextual information to achieve strong connectivity between segments.

[0006] ② Generate accurate and sensitive pseudo queries: In existing algorithms for zero-shot natural language video action localization, the generated pseudo queries are often too isolated and difficult to adapt to the language framework and query requirements in open-world scenarios, failing to meet the standards for generating complex, high-quality pseudo queries. In addition, existing methods tend to focus on visually significant scene changes while ignoring those subtle but semantically important changes, which leads to inaccurate segmentation of some cases and generates query text that does not match the video content.

[0007] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a cross-modal power video positioning method, system, electronic device and storage medium.

[0009] In a first aspect, the present invention provides a cross-modal power video positioning method, and the technical solution of the method is as follows:

[0010] Perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video in turn to obtain at least one target pseudo-supervision pair corresponding to each sample power video; wherein each target pseudo-supervision pair includes: a corresponding sample power video, a pseudo-query text and the start and end time of the pseudo-query text in the sample power video;

[0011] Based on the target pseudo-supervision pair corresponding to each sample power video, a preset cross-modal power video localization model is trained to obtain a trained cross-modal power video localization model;

[0012] The power video to be tested and the target query text are input into the trained cross-modal power video positioning model to obtain the video positioning result of the power video to be tested.

[0013] The beneficial effects of a cross-modal power video positioning method of the present invention are as follows:

[0014] The method of the present invention generates temporal dynamic pseudo-supervision pairs with boundary perception features through visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation, and trains a fully supervised cross-modal power video localization model based on the generated pseudo-supervision pairs, thereby achieving robust and efficient zero-sample cross-modal power abnormal behavior localization and improving the accuracy and efficiency of cross-modal power video localization.

[0015] Based on the above solution, the cross-modal power video positioning method of the present invention can also be improved as follows.

[0016] In an optional manner, the step of performing visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on any sample power video in sequence to obtain at least one target pseudo-supervision pair corresponding to any sample power video includes:

[0017] Performing visual boundary segmentation on any of the sample power videos to obtain a plurality of visual segments, and performing semantic boundary segmentation on each visual segment to obtain a plurality of atomic segments corresponding to each visual segment;

[0018] By utilizing the context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment, a target pseudo-supervision pair corresponding to any sample power video is determined.

[0019] In an optional manner, the step of performing semantic boundary segmentation on any visual segment to obtain a plurality of atomic segments corresponding to the any visual segment includes:

[0020] Extracting at least one key frame in any one of the visual segments, and obtaining a description text corresponding to each key frame in any one of the visual segments;

[0021] The description text corresponding to any visual segment and each key frame in the any visual segment is input into the large language model, and combined with the prompt words, a plurality of atomic segments corresponding to the any visual segment are obtained.

[0022] In an optional manner, the description text includes: title text and label text; the step of obtaining the description text corresponding to each key frame in any one of the visual segments includes:

[0023] The image caption model is used to obtain the title text corresponding to each key frame in any visual segment, and the multimodal large model is used to obtain the label text corresponding to each key frame in any visual segment.

[0024] In an optional manner, the step of determining a target pseudo-supervisory pair corresponding to any sample power video by using a context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment includes:

[0025] Using the contextual dynamic aggregation method of the large language model and combining multiple atomic segments corresponding to each visual segment, each initial pseudo-supervisory pair corresponding to any sample power video is obtained;

[0026] Using a large language model, a plurality of atomic segments corresponding to each visual segment in any one of the sample power videos, and each initial pseudo-supervisory pair corresponding to any one of the sample power videos, a score of each initial pseudo-supervisory pair corresponding to any one of the sample power videos is obtained;

[0027] An initial pseudo-supervisory pair with a score higher than a target threshold is determined as a target pseudo-supervisory pair corresponding to any one of the sample power videos.

[0028] In a second aspect, the present invention provides a cross-modal power video positioning system, the technical solution of which is as follows:

[0029] Includes: generation module, training module and detection module;

[0030] The generation module is used to perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video in sequence, and obtain at least one target pseudo-supervision pair corresponding to each sample power video; wherein each target pseudo-supervision pair includes: a corresponding sample power video, a pseudo-query text, and a start and end time of the pseudo-query text in the sample power video;

[0031] The training module is used to: train a preset cross-modal power video positioning model based on the target pseudo-supervision pair corresponding to each sample power video to obtain a trained cross-modal power video positioning model;

[0032] The detection module is used to: input the power video to be tested and the target query text into the trained cross-modal power video positioning model to obtain the video positioning result of the power video to be tested.

[0033] The beneficial effects of a cross-modal power video positioning system of the present invention are as follows:

[0034] The system of the present invention generates temporal dynamic pseudo-supervision pairs with boundary perception features through visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation, and trains a fully supervised cross-modal power video localization model based on the generated pseudo-supervision pairs, thereby achieving robust and efficient zero-sample cross-modal power abnormal behavior localization and improving the accuracy and efficiency of cross-modal power video localization.

[0035] Based on the above solution, the cross-modal power video positioning system of the present invention can also be improved as follows.

[0036] In an optional manner, the generating module is specifically used for:

[0037] Perform visual boundary segmentation on any sample power video to obtain multiple visual segments, and perform semantic boundary segmentation on each visual segment to obtain multiple atomic segments corresponding to each visual segment;

[0038] By utilizing the context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment, a target pseudo-supervision pair corresponding to any sample power video is determined.

[0039] In an optional manner, the generating module is specifically used for:

[0040] Extracting at least one key frame in any one of the visual segments, and obtaining a description text corresponding to each key frame in any one of the visual segments;

[0041] The description text corresponding to any visual segment and each key frame in the any visual segment is input into the large language model, and combined with the prompt words, a plurality of atomic segments corresponding to the any visual segment are obtained.

[0042] In a third aspect, a technical solution of an electronic device of the present invention is as follows:

[0043] The invention comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the cross-modal power video positioning method of the present invention are implemented.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0045] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the cross-modal power video positioning method of the present invention.

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0048] Figure 1 A schematic diagram of a flow chart of an embodiment of a cross-modal power video positioning method of the present invention;

[0049] Figure 2 It is a schematic diagram of the overall principle of the cross-modal power video positioning method;

[0050] Figure 3 It is a structural schematic diagram of an embodiment of a cross-modal power video positioning system of the present invention;

[0051] Figure 4 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0052] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0053] Figure 1 A flow chart of an embodiment of a cross-modal power video positioning method provided by the present invention is shown, and the cross-modal power video positioning method can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the cross-modal power video positioning method by calling computer-readable instructions stored in a memory through a processor. As Figure 1 As shown, the following steps are included:

[0054] S1. Perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video in turn to obtain at least one target pseudo-supervision pair corresponding to each sample power video.

[0055] Among them, the sample power video is a sample video of the power production construction scene used for the training of the cross-modal power video positioning model. The sample power video contains abnormal behavior by default. Visual boundary segmentation refers to the process of dividing the video content into multiple segments based on significant changes in visual features (such as object edges, motion, color, texture, etc.). The segmentation boundary is usually defined by discontinuities or mutations in the video frame, such as mutations in grayscale values, mutations in colors, mutations in texture structures, etc. It is assumed that changes in video content correspond to significant changes in visual features. Based on this assumption, the sample power video is divided into multiple visual segments by using a clipping tool that segments according to pixel changes. Video semantic segmentation is used to accurately understand each frame of the video and assign correct semantic labels, and semantic boundary segmentation further divides the boundaries of semantic changes. Each target pseudo-supervision pair includes: the corresponding sample power video, the pseudo query text, and the start and end time of the pseudo query text in the sample power video.

[0056] S2. Based on the target pseudo-supervision pair corresponding to each sample power video, the preset cross-modal power video positioning model is trained to obtain a trained cross-modal power video positioning model.

[0057] Among them, the preset cross-modal power video positioning model is a conventional cross-modal video positioning model, which is used to detect abnormal events and personnel behaviors, and the specific structure of the model is not restricted here.

[0058] S3. Input the power video to be tested and the target query text into the trained cross-modal power video positioning model to obtain the video positioning result of the power video to be tested.

[0059] The power video to be tested is the power video to be video-located in this embodiment. The target query text refers to: the text description entered by the user in order to locate the relevant target segment from the power video to be tested. The text description usually contains keywords, phrases or sentences related to the video content, which are used to guide the model to search and match in the video. The video positioning result refers to: the target segment related to the query located by the model from the power video to be tested according to the target query text. The target segment is usually presented in the form of a video frame sequence, a time period or a specific area in the video.

[0060] In an optional manner, the step of performing visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on any sample power video in sequence to obtain at least one target pseudo-supervision pair corresponding to any sample power video includes:

[0061] Perform visual boundary segmentation on any sample power video to obtain a plurality of visual segments, and perform semantic boundary segmentation on each visual segment to obtain a plurality of atomic segments corresponding to each visual segment.

[0062] The sample power video includes N video frames, and the sample power video is represented as: V = {F1, ..., F N Perform visual boundary segmentation on any sample power video to obtain multiple visual segments: i represents the number of visual segments, represents the i-th visual segment, each visual segment includes at least one video frame. The multiple atomic segments corresponding to any visual segment are expressed as: It is represented as the kth atomic fragment corresponding to the i-th visual fragment.

[0063] By utilizing the context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment, a target pseudo-supervision pair corresponding to any sample power video is determined.

[0064] Among them, the number of target pseudo-supervision pairs is at least one, and the target pseudo-supervision pair set T is expressed as: K represents the number of target pseudo-supervision pairs, V j represents the sample power video corresponding to the jth target pseudo-supervision pair, Q jis the jth pseudo query text, Indicates that the jth pseudo query text corresponds to V j The start and end time in ( Indicates the starting time, Indicates the end time).

[0065] In an optional manner, the step of performing semantic boundary segmentation on any visual segment to obtain a plurality of atomic segments corresponding to the any visual segment includes:

[0066] At least one key frame in any visual segment is extracted, and a description text corresponding to each key frame in the any visual segment is obtained.

[0067] Among them, due to the visual redundancy in the video, in order to improve processing efficiency and accurately capture the main content of each visual segment, key frame extraction is performed on the visual segment. This process comprehensively considers multiple factors such as LUV color space, brightness, contrast and blur, and uses the k-means clustering algorithm to identify and extract key frames.

[0068] The description text corresponding to any visual segment and each key frame in the any visual segment is input into the large language model, and combined with the prompt words, a plurality of atomic segments corresponding to the any visual segment are obtained.

[0069] In order to further capture the semantic changes that are ignored by relying solely on visual boundary segmentation, more detailed segmentation is required based on the changes in semantic content. Visual boundary segmentation may ignore some important semantic boundaries, so it is necessary to segment these visual segments. Semantic boundary segmentation is performed to obtain more accurate content representation. Visual fragments that are ignored in visual boundary segmentation It can be described as: ∥ indicates a semantic boundary within the visual boundary. In order to achieve the goal of semantic boundary segmentation, the visual fragments and their corresponding keyframe descriptions (description texts) are input into the large language model to infer all semantically different activities for each visual fragment. Through carefully designed prompts, multiple instructions and criteria are set to help LLM better understand the task requirements. Specifically, the prompts contain clear instructions and evaluation criteria to ensure that LLM can accurately identify the time nodes of semantic changes in the keyframe representation. Subsequently, LLM performs semantic boundary segmentation according to the time nodes of changes in the keyframe representation to obtain the segmented atomic fragment Atom:

[0070]

[0071] In an optional manner, the description text includes: title text and label text; the step of obtaining the description text corresponding to each key frame in any one of the visual segments includes:

[0072] The image caption model is used to obtain the title text corresponding to each key frame in any visual segment, and the multimodal large model is used to obtain the label text corresponding to each key frame in any visual segment.

[0073] Among them, in order to generate key frame representations that can accurately describe the video content, a multi-granularity representation method is adopted to ensure the comprehensiveness and accuracy of the description. The title text is a coarse-grained text that is used to summarize the main content described in the image. These contents (subtitles) provide a high-level understanding of the entire scene and help to quickly grasp the core information of each key frame; through the target detection and instance annotation model in the multimodal large model, the detailed elements (label text) in each key frame are identified and annotated, including: labels of multiple dimensions such as objects, actions, and attributes, so as to achieve a detailed characterization of each key frame. The description text for each key frame can be expressed as: Caption (n) Indicates the title text of the nth key frame, Tag (n) Label text representing the nth keyframe.

[0074] In an optional manner, the step of determining a target pseudo-supervisory pair corresponding to any sample power video by using a context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment includes:

[0075] By utilizing the contextual dynamic aggregation method of the large language model and combining multiple atomic segments corresponding to each visual segment, each initial pseudo-supervisory pair corresponding to any sample power video is obtained.

[0076] Among them, the prompt word template for the large language model initialization context dynamic aggregation task. The template first gives <example and video> and the large language model as the role definition of the video analyst, and then gives the task definition as merging and summarizing the initial atomic segments of the video and the description of each segment, and generates a pseudo-supervisory pair (visual segment and corresponding description) as the initial pseudo-supervisory pair R of context dynamic aggregation. init =LLM(V sb ); It represents the atomic segment obtained by the aforementioned visual boundary segmentation and semantic boundary segmentation for each visual segment corresponding to the sample power video.

[0077] It should be noted that the evaluation criteria for the dynamic aggregation results of the large language model context set in this embodiment are: ① Non-redundancy: Ensure that each segment describes a unique activity and integrate segments involving the same activity. ② Singularity: Each segment should only describe one activity, and avoid mixing multiple activities in a single segment. ③ Completeness: Adjust the segments according to the overall information of the video, unify the segment descriptions, and emphasize the time sequence and logical relationships. ④ Non-fragmentation: Avoid short segments (less than 5 seconds) and ensure that each segment is not fragmented content. ⑤ Accuracy: Based on the global information of the video, ensure that the description of each segment is accurate and no uncertainty is allowed.

[0078] A score of each initial pseudo-supervisory pair corresponding to any sample power video is obtained by using a large language model, a plurality of atomic segments corresponding to each visual segment in any sample power video, and each initial pseudo-supervisory pair corresponding to any sample power video.

[0079] Among them, multiple atomic fragments corresponding to each visual fragment in any sample power video and any initial pseudo-supervisory pair corresponding to any sample power video are input into the large language model to obtain the score of any initial pseudo-supervisory pair, and the process is repeated until the score of each initial pseudo-supervisory pair corresponding to any sample power video is obtained.

[0080] In order to ensure the accuracy of the large language model's dynamic aggregation of visual fragments, this embodiment requires the large language model to generate each initial pseudo-supervisory pair R init , based on more than five criteria (such as accuracy, consistency, relevance, completeness and interpretability, etc.), namely: S = LLM (V sb ,R init ), S represents the score of the initial pseudo-supervisory pair.

[0081] An initial pseudo-supervisory pair with a score higher than a target threshold is determined as a target pseudo-supervisory pair corresponding to any one of the sample power videos.

[0082] The value of the target threshold t can be set according to the actual situation and is not limited here. The number of target pseudo-supervision pairs is at least one.

[0083] It should be noted that Figure 2 The overall principle diagram of this embodiment is shown. Loss function of the cross-modal power video positioning model for: L is the loss function for a single target supervision pair.

[0084] As shown in Table 1, the positioning accuracy and efficiency comparison results are obtained by comparing the positioning performance of the cross-modal power video positioning model trained by the method in this embodiment with the international leading similar models. It can be seen from Table 1 that compared with the similar comparison models, the positioning accuracy of the cross-modal power video positioning model in this embodiment is significantly superior.

[0085] Table 1:

[0086]

[0087] This embodiment first performs visual boundary segmentation and semantic boundary segmentation, specifically segments the video into visual boundaries according to the change of visual features, then extracts key frames from each visual segment, generates coarse-grained titles and fine-grained labels as the description text of the key frame, inputs the description text of the visual segment and its corresponding key frame into a large language model, and uses the large language model to further divide the semantic boundary according to the semantic change in the key frame pair representation, and obtains the atomic segment of the video. This embodiment then generates pseudo-supervisory pairs based on contextual dynamic aggregation, specifically inputs the atomic segments of the entire video and their description text into a large language model, analyzes the global information of the video based on the global information generation of the video, and performs contextual aggregation; by checking the time and logical relationship between the atomic segments, LLM merges semantically similar segments, summarizes and generates queries for each merged segment. Through this process, the present invention realizes contextual aggregation, obtains a non-redundant final segment with a new context-related query, and thus generates a high-quality pseudo-supervisory pairing. At the same time, this embodiment sets multiple instructions and standards in the prompt to guide the large language model to self-scoring, which promotes the large language model to better understand the task. Finally, this embodiment trains a fully supervised cross-modal power video localization model based on the generated pseudo-supervisory pairs, achieving robust and efficient zero-sample cross-modal video localization.

[0088] The technical solution of this embodiment generates temporal dynamic pseudo-supervision pairs with boundary perception features through visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation, and trains a fully supervised cross-modal power video localization model based on the generated pseudo-supervision pairs, thereby achieving robust and efficient zero-sample cross-modal power abnormal behavior localization and improving the accuracy and efficiency of cross-modal power video localization.

[0089] Figure 3 FIG. 2 is a schematic diagram showing a structure of an embodiment of a cross-modal power video positioning system 200 provided by the present invention. Figure 3 As shown, the system 200 includes: a generation module 210, a training module 220 and a detection module 230;

[0090] The generation module 210 is used to perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video in sequence, and obtain at least one target pseudo-supervision pair corresponding to each sample power video; wherein each target pseudo-supervision pair includes: a corresponding sample power video, a pseudo-query text, and a start and end time of the pseudo-query text in the sample power video;

[0091] The training module 220 is used to: train a preset cross-modal power video positioning model based on the target pseudo-supervision pair corresponding to each sample power video to obtain a trained cross-modal power video positioning model;

[0092] The detection module 230 is used to: input the power video to be tested and the target query text into the trained cross-modal power video positioning model to obtain the video positioning result of the power video to be tested.

[0093] In an optional manner, the generating module 210 is specifically used for:

[0094] Perform visual boundary segmentation on any sample power video to obtain multiple visual segments, and perform semantic boundary segmentation on each visual segment to obtain multiple atomic segments corresponding to each visual segment;

[0095] By utilizing the context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment, a target pseudo-supervision pair corresponding to any sample power video is determined.

[0096] In an optional manner, the generating module 210 is specifically used for:

[0097] Extracting at least one key frame in any one of the visual segments, and obtaining a description text corresponding to each key frame in any one of the visual segments;

[0098] The description text corresponding to any visual segment and each key frame in the any visual segment is input into the large language model, and combined with the prompt words, a plurality of atomic segments corresponding to the any visual segment are obtained.

[0099] In an optional manner, the description text includes: title text and tag text; the generation module 210 is specifically used to:

[0100] The image caption model is used to obtain the title text corresponding to each key frame in any visual segment, and the multimodal large model is used to obtain the label text corresponding to each key frame in any visual segment.

[0101] In an optional manner, the generating module 210 is specifically used for:

[0102] Using the contextual dynamic aggregation method of the large language model and combining multiple atomic segments corresponding to each visual segment, each initial pseudo-supervisory pair corresponding to any sample power video is obtained;

[0103] Using a large language model, a plurality of atomic segments corresponding to each visual segment in any one of the sample power videos, and each initial pseudo-supervisory pair corresponding to any one of the sample power videos, a score of each initial pseudo-supervisory pair corresponding to any one of the sample power videos is obtained;

[0104] An initial pseudo-supervisory pair with a score higher than a target threshold is determined as a target pseudo-supervisory pair corresponding to any one of the sample power videos.

[0105] It should be noted that the beneficial effects of the cross-modal power video positioning system provided in the above-mentioned embodiment are the same as the beneficial effects of the cross-modal power video positioning method, which will not be described in detail here. In addition, when the system provided in the above-mentioned embodiment realizes its functions, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above-mentioned embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

[0106] Among them, the cross-modal power video positioning system of the present invention can be a computer program (including program code) running in a computer device. For example, the cross-modal power video positioning system of the present invention is an application software that can be used to execute the corresponding steps in the cross-modal power video positioning method of the present invention.

[0107] In some embodiments, the cross-modal power video positioning system of the present invention can be implemented by a combination of software and hardware. As an example, the cross-modal power video positioning system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the cross-modal power video positioning method of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0108] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0109] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned cross-modal power video positioning methods is implemented. That is to say, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the cross-modal power video positioning method shown in any embodiment of the present invention by calling the computer program.

[0110] In an alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0111] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0112] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 In the figure, only one thick line is used to represent the bus 4002, but this does not mean that there is only one bus or one type of bus.

[0113] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0114] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0115] Among them, the electronic device can also be a terminal device, and the terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0116] It should be noted that Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned cross-modal power video positioning methods is implemented.

[0118] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0119] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the above-mentioned cross-modal power video positioning method.

[0120] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0121] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0123] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0124] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0125] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0126] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0127] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A cross-modal power video positioning method, characterized in that: include: Perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video in turn to obtain at least one target pseudo-supervision pair corresponding to each sample power video; wherein each target pseudo-supervision pair includes: a corresponding sample power video, a pseudo-query text and the start and end time of the pseudo-query text in the sample power video; Based on the target pseudo-supervision pair corresponding to each sample power video, a preset cross-modal power video localization model is trained to obtain a trained cross-modal power video localization model; The power video to be tested and the target query text are input into the trained cross-modal power video positioning model to obtain the video positioning result of the power video to be tested.

2. The cross-modal power video positioning method according to claim 1, characterized in that: The step of sequentially performing visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on any sample power video to obtain at least one target pseudo-supervision pair corresponding to any sample power video includes: Performing visual boundary segmentation on any sample power video to obtain a plurality of visual segments, and performing semantic boundary segmentation on each visual segment to obtain a plurality of atomic segments corresponding to each visual segment; By utilizing the context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment, a target pseudo-supervision pair corresponding to any sample power video is determined.

3. The cross-modal power video positioning method according to claim 2, characterized in that: The step of performing semantic boundary segmentation on any visual segment to obtain a plurality of atomic segments corresponding to the any visual segment comprises: Extracting at least one key frame in any one of the visual segments, and obtaining a description text corresponding to each key frame in any one of the visual segments; The description text corresponding to any visual segment and each key frame in the visual segment is input into the large language model, and combined with the prompt words, a plurality of atomic segments corresponding to the visual segment are obtained.

4. The cross-modal power video positioning method according to claim 3, characterized in that: The description text includes: title text and label text; the step of obtaining the description text corresponding to each key frame in any one of the visual segments includes: The image caption model is used to obtain the title text corresponding to each key frame in any visual segment, and the multimodal large model is used to obtain the label text corresponding to each key frame in any visual segment.

5. The cross-modal power video positioning method according to any one of claims 2 to 4, characterized in that: The step of determining a target pseudo-supervision pair corresponding to any sample power video by using a context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment includes: Using the contextual dynamic aggregation method of the large language model and combining multiple atomic segments corresponding to each visual segment, each initial pseudo-supervisory pair corresponding to any sample power video is obtained; Using a large language model, a plurality of atomic segments corresponding to each visual segment in any one of the sample power videos, and each initial pseudo-supervisory pair corresponding to any one of the sample power videos, a score of each initial pseudo-supervisory pair corresponding to any one of the sample power videos is obtained; An initial pseudo-supervisory pair with a score higher than a target threshold is determined as a target pseudo-supervisory pair corresponding to any one of the sample power videos.

6. A cross-modal power video positioning system, characterized in that: include: Generation module, training module and detection module; The generation module is used to perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video in sequence, and obtain at least one target pseudo-supervision pair corresponding to each sample power video; wherein each target pseudo-supervision pair includes: a corresponding sample power video, a pseudo-query text, and a start and end time of the pseudo-query text in the sample power video; The training module is used to: train a preset cross-modal power video positioning model based on the target pseudo-supervision pair corresponding to each sample power video to obtain a trained cross-modal power video positioning model; The detection module is used to: input the power video to be tested and the target query text into the trained cross-modal power video positioning model to obtain the video positioning result of the power video to be tested.

7. The cross-modal power video positioning system according to claim 6, characterized in that: The generation module is specifically used for: Perform visual boundary segmentation on any sample power video to obtain multiple visual segments, and perform semantic boundary segmentation on each visual segment to obtain multiple atomic segments corresponding to each visual segment; By utilizing the context dynamic aggregation method and combining multiple atomic segments corresponding to each visual segment, a target pseudo-supervision pair corresponding to any sample power video is determined.

8. The cross-modal power video positioning system according to claim 7, characterized in that: The generation module is specifically used for: Extracting at least one key frame in any one of the visual segments, and obtaining a description text corresponding to each key frame in any one of the visual segments; The description text corresponding to any visual segment and each key frame in the visual segment is input into the large language model, and combined with the prompt words, a plurality of atomic segments corresponding to the visual segment are obtained.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the cross-modal power video positioning method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: At least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the cross-modal power video positioning method as described in any one of claims 1 to 5.

Images