Cross-modal power video positioning method, system, electronic device and storage medium

Abstract

The application relates to the technical field of video understanding, and specifically discloses a cross-modal power video positioning method and system, an electronic device and a storage medium, which comprises the following steps: sequentially performing visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video to obtain at least one target pseudo-supervision pair corresponding to each sample power video; each target pseudo-supervision pair comprises a corresponding sample power video, a pseudo query text and start and end times of the pseudo query text in the sample power video; based on the target pseudo-supervision pair corresponding to each sample power video, a preset cross-modal power video positioning model is trained to obtain a trained cross-modal power video positioning model; and the to-be-tested power video and the query text are input into the trained cross-modal power video positioning model to obtain a video positioning result. The application realizes robust and efficient zero-shot cross-modal power abnormal behavior positioning, and improves the accuracy and efficiency of cross-modal power video positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video understanding, and in particular to a cross-modal power video positioning method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the vigorous development of urban economy, the scale and demand of power engineering construction are growing, and the perception monitoring of safety hazards in power production construction scenes has put forward more strict and urgent requirements. As a key means of perception detection, video monitoring plays an increasingly important role in real-time decision-making and accurate early warning, and needs to be able to quickly locate specific actions or events in unedited video content based on natural language queries. Therefore, cross-modal video positioning technology becomes particularly important.

[0003] Cross-modal video positioning is a crucial and challenging task in the field of video understanding, and its core goal is to accurately locate specific video moments in untrimmed videos according to corresponding language queries. For example, given text queries such as "non-standard safety belt wearing" and "non-standard wearing of labor protection articles", this technology can accurately identify and locate the specific timing interval of these rule-breaking operations in the video. However, existing natural language video action positioning technologies mostly rely on complete supervision or weak supervision for large-scale model training, which requires manual annotation of the precise timing constraints of each action category, consuming a large amount of manual annotation cost, and being difficult to adapt to complex and variable actual situations, diversified rule-breaking operations, and the demand of massive video monitoring. In contrast, zero-shot natural language video positioning technology does not require any manual annotation and can directly generate pseudo queries and corresponding video moment pairs to train the positioning model, showing higher scalability and cost-effectiveness.

[0004] Natural language video action positioning research under the zero-shot setting has made significant progress in the traditional video action positioning field and has shown excellent scalability to adapt to the growing perception detection needs. However, research in this field still faces two major challenges:

[0005] ① Enhance inter-fragment connectivity and integrate context information: Current research methods often ignore the importance of context and treat visual fragments as independent units. In video sequences, although the semantic context remains consistent, segmentation methods that rely only on visual features can lead to fragmentation of visual fragments. In addition, the isolation between fragments results in the lack of context information, which not only limits the ability to infer entire dynamic activity sequences from static image captions, but also affects the ability to establish temporal relationships between pseudo queries. To overcome this challenge, research needs to develop methods that can identify and integrate context information to achieve strong connectivity between fragments.

[0006] ②Generating accurate and sensitive pseudo queries: In existing algorithms for zero-shot natural language video action localization, the generated pseudo queries are often too isolated and difficult to adapt to the language framework and query requirements in open-world scenarios, failing to meet the standards for generating complex and high-quality pseudo queries. In addition, existing methods tend to focus on visually significant scene changes, while ignoring subtle but semantically important changes, which leads to inaccurate segmentation of certain situations and further generates query texts that do not match the video content.

[0007] Therefore, there is an urgent need to provide a technical solution to solve the above problems. SUMMARY

[0008] To solve the above technical problems, the present application provides a cross-modal power video localization method, system, electronic device and storage medium.

[0009] In a first aspect, the present application provides a cross-modal power video localization method, and the technical solution of the method is as follows:

[0010] The visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation are sequentially performed on each sample power video to obtain at least one target pseudo supervision pair corresponding to each sample power video; wherein each target pseudo supervision pair includes: a corresponding sample power video, a pseudo query text and a start and end time of the pseudo query text in the sample power video;

[0011] Based on the target pseudo supervision pair corresponding to each sample power video, a pre-set cross-modal power video localization model is trained to obtain a trained cross-modal power video localization model;

[0012] The test power video and the target query text are input into the trained cross-modal power video localization model to obtain the video localization result of the test power video.

[0013] The cross-modal power video localization method of the present application has the following beneficial effects:

[0014] The method of the present application generates time dynamic pseudo supervision pairs with boundary perception features through visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation, and trains a fully supervised cross-modal power video localization model based on the generated pseudo supervision pairs, achieving robust and efficient zero-shot cross-modal power abnormal behavior localization and improving the accuracy and efficiency of cross-modal power video localization.

[0015] Based on the above-mentioned scheme, the cross-modal power video localization method of the present application can also be improved as follows.

[0016] In an optional manner, the step of determining the at least one target pseudo-supervision pair corresponding to the any sample power video in sequence through visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation includes:

[0017] Segmenting the any sample power video through visual boundary segmentation to obtain a plurality of visual segments, and segmenting each visual segment through semantic boundary segmentation to obtain a plurality of atomic segments corresponding to each visual segment;

[0018] Determining the target pseudo-supervision pair corresponding to the any sample power video through the context dynamic aggregation manner and in combination with the plurality of atomic segments corresponding to each visual segment.

[0019] In an optional manner, the step of segmenting the any visual segment through semantic boundary segmentation to obtain a plurality of atomic segments corresponding to the any visual segment includes:

[0020] Extracting at least one key frame in the any visual segment, and obtaining description text corresponding to each key frame in the any visual segment;

[0021] Inputting the any visual segment and the description text corresponding to each key frame in the any visual segment into a large language model, and obtaining a plurality of atomic segments corresponding to the any visual segment in combination with a prompt word.

[0022] In an optional manner, the description text includes title text and label text, and the step of obtaining the description text corresponding to each key frame in the any visual segment includes:

[0023] Obtaining the title text corresponding to each key frame in the any visual segment through an image caption model, and obtaining the label text corresponding to each key frame in the any visual segment through a multi-modal large model.

[0024] In an optional manner, the step of determining the target pseudo-supervision pair corresponding to the any sample power video through the context dynamic aggregation manner and in combination with the plurality of atomic segments corresponding to each visual segment includes:

[0025] Obtaining each initial pseudo-supervision pair corresponding to the any sample power video through the context dynamic aggregation manner of a large language model and in combination with the plurality of atomic segments corresponding to each visual segment;

[0026] Obtaining a score of each initial pseudo-supervision pair corresponding to the any sample power video through the large language model, the plurality of atomic segments corresponding to each visual segment in the any sample power video, and each initial pseudo-supervision pair corresponding to the any sample power video;

[0027] The initial pseudo-supervision pair with a score higher than the target threshold is determined as the target pseudo-supervision pair corresponding to the any sample power video.

[0028] In a second aspect, the present application provides a cross-modal power video positioning system, and the technical scheme of the system is as follows:

[0029] The system comprises a generation module, a training module and a detection module.

[0030] The generation module is configured to sequentially perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video to obtain at least one target pseudo-supervision pair corresponding to each sample power video; wherein each target pseudo-supervision pair comprises a corresponding sample power video, a pseudo query text and a start and end time of the pseudo query text in the sample power video.

[0031] The training module is configured to train a preset cross-modal power video positioning model based on the target pseudo-supervision pair corresponding to each sample power video to obtain a trained cross-modal power video positioning model.

[0032] The detection module is configured to input a to-be-tested power video and a target query text into the trained cross-modal power video positioning model to obtain a video positioning result of the to-be-tested power video.

[0033] The cross-modal power video positioning system of the present application has the following beneficial effects:

[0034] The system of the present application generates time dynamic pseudo-supervision pairs with boundary perception features through visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation, and trains a fully supervised cross-modal power video positioning model based on the generated pseudo-supervision pairs, thereby realizing robust and efficient zero-shot cross-modal power abnormal behavior positioning and improving the accuracy and efficiency of cross-modal power video positioning.

[0035] On the basis of the above-mentioned scheme, the cross-modal power video positioning system of the present application can be further improved as follows.

[0036] In an optional manner, the generation module is specifically configured to:

[0037] The visual boundary segmentation is performed on any sample power video to obtain a plurality of visual segments, and the semantic boundary segmentation is performed on each visual segment to obtain a plurality of atomic segments corresponding to each visual segment.

[0038] The target pseudo-supervision pair corresponding to the any sample power video is determined by using the context dynamic aggregation and combining the plurality of atomic segments corresponding to each visual segment.

[0039] In an optional manner, the generation module is specifically configured to:

[0040] extracting at least one key frame in the any visual segment, and obtaining description text corresponding to each key frame in the any visual segment;

[0041] inputting the any visual segment and the description text corresponding to each key frame in the any visual segment into a large language model, and obtaining a plurality of atomic segments corresponding to the any visual segment in combination with a prompt word.

[0042] In a third aspect, a technical solution of an electronic device of the present application is as follows:

[0043] comprising a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor implements the steps of the cross-modal power video positioning method of the present application when executing the program.

[0044] In a fourth aspect, a technical solution of a computer readable storage medium provided by the present application is as follows:

[0045] The computer readable storage medium stores instructions, and when the computer readable storage medium reads the instructions, the computer readable storage medium executes the steps of the cross-modal power video positioning method of the present application.

[0046] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the present application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings are only used to illustrate the embodiments and are not considered as limiting the present application. Moreover, the same reference signs are used to represent the same components throughout the drawings. In the drawings:

[0048] Figure 1 a flowchart of an embodiment of a cross-modal power video positioning method of the present application;

[0049] Figure 2 a schematic diagram of the overall principle of the cross-modal power video positioning method;

[0050] Figure 3 a structural schematic diagram of an embodiment of a cross-modal power video positioning system of the present application;

[0051] Figure 4 a structural schematic diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0052] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein.

[0053] Figure 1 A flowchart of an embodiment of a cross-modal power video positioning method provided by the present application is shown, which can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the cross-modal power video positioning method by calling computer-readable instructions stored in the memory through the processor. As shown in Figure 1 The steps include:

[0054] S1. Sequentially performing visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video to obtain at least one target pseudo-supervised pair corresponding to each sample power video.

[0055] The sample power video is a sample video of a power production construction scene used for training of a cross-modal power video positioning model, and the sample power video by default contains abnormal behavior. Visual boundary segmentation refers to a process of dividing video content into multiple segments according to significant changes in visual features such as object edges, motion, color, texture, etc. The segmentation boundary is usually defined by discontinuity or abrupt change in the video frame, such as abrupt change in gray value, abrupt change in color, abrupt change in texture structure, etc. It is assumed that the change in video content corresponds to the significant change in visual features. Based on this assumption, the sample power video is divided into multiple visual segments by using a clip tool for segmentation according to pixel changes. Video semantic segmentation is used to accurately understand each frame of image in the video and assign correct semantic labels, and semantic boundary segmentation further divides the boundary of semantic changes. Each target pseudo-supervised pair includes the corresponding sample power video, the pseudo query text and the start and end time of the pseudo query text in the sample power video.

[0056] S2. Training a preset cross-modal power video positioning model based on the target pseudo-supervised pair corresponding to each sample power video to obtain a trained cross-modal power video positioning model.

[0057] The preset cross-modal power video positioning model is a conventional cross-modal video positioning model for detecting abnormal events and personnel behaviors, and the specific structure of the model is not limited herein.

[0058] S3, inputting the to-be-tested power video and the target query text into the trained cross-modal power video positioning model to obtain a video positioning result of the to-be-tested power video.

[0059] The to-be-tested power video is a power video to be subjected to video positioning in the embodiment. The target query text refers to a text description input by a user in order to locate a relevant target segment from the to-be-tested power video. The text description usually contains keywords, phrases or sentences related to the video content, and is used to guide the model to search and match in the video. The video positioning result refers to a target segment related to the query located by the model from the to-be-tested power video according to the target query text. The target segment is usually in the form of a video frame sequence, a time period or a specific region in the video.

[0060] In an optional manner, the step of sequentially performing visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video to obtain at least one target pseudo-supervised pair corresponding to the sample power video comprises:

[0061] The visual boundary segmentation is performed on the sample power video to obtain a plurality of visual segments, and the semantic boundary segmentation is performed on each visual segment to obtain a plurality of atomic segments corresponding to each visual segment.

[0062] The sample power video contains N video frames, and the sample power video is represented as: V = {F1,..., FN}. N The visual boundary segmentation is performed on the sample power video to obtain a plurality of visual segments: i represents the number of visual segments, The i-th visual segment, each visual segment includes at least one video frame. The plurality of atomic segments corresponding to any visual segment is represented as: The k-th atomic segment corresponding to the i-th visual segment is represented as:

[0063] The context dynamic aggregation is used to determine the target pseudo-supervised pair corresponding to the sample power video in combination with the plurality of atomic segments corresponding to each visual segment.

[0064] The number of target pseudo-supervised pairs is at least one, and the set of target pseudo-supervised pairs T is represented as: K represents the number of target pseudo-supervised pairs, V j The j-th target pseudo-supervised pair corresponds to a sample power video, and Q jwherein the jth pseudo query text, denotes the start and end time of the jth pseudo query text in V j denotes the start time, denotes the end time. denotes the end time.

[0065] In an optional manner, the step of performing semantic boundary segmentation on any visual segment to obtain a plurality of atomic segments corresponding to the any visual segment comprises:

[0066] Extracting at least one key frame in any visual segment, and obtaining a description text corresponding to each key frame in the any visual segment.

[0067] Wherein, due to the existence of visual redundancy in the video, in order to improve the processing efficiency and accurately capture the main content of each visual segment, key frame extraction is performed on the visual segment. This process considers a variety of factors such as LUV color space, brightness, contrast and blurriness, and uses k-means clustering algorithm to identify and extract key frames.

[0068] Inputting the any visual segment and the description text corresponding to each key frame in the any visual segment into a large language model, and combining a prompt word, to obtain a plurality of atomic segments corresponding to the any visual segment.

[0069] Wherein, in order to further capture the semantic changes ignored by the visual boundary segmentation which only relies on visual boundary segmentation, more detailed segmentation is needed according to the changes of semantic content. Visual boundary segmentation may ignore some important semantic boundaries, so it is necessary to perform semantic boundary segmentation on these visual segments to obtain more accurate content representation. The visual segments ignored in visual boundary segmentation can be described as: denotes the semantic boundary within the visual boundary. In order to achieve the goal of semantic boundary segmentation, the visual segment and its corresponding key frame description (description text) are input into the large language model to infer all semantically different activities of each visual segment. A well-designed prompt word sets multiple instructions and standards to help the LLM better understand the task requirements. Specifically, the prompt word contains clear instructions and evaluation standards to ensure that the LLM can accurately identify the time nodes of semantic changes in the key frame representation. Subsequently, the LLM performs semantic boundary segmentation according to the time nodes of changes in the key frame representation to obtain the segmented atomic segments Atom:

[0070]

[0071] In an optional manner, the description text includes: title text and label text; the step of obtaining the description text corresponding to each key frame in any visual segment includes:

[0072] Using the image captioning model, the title text corresponding to each key frame in any visual segment is obtained, and the label text corresponding to each key frame in any visual segment is obtained using the multi-modal large model.

[0073] Wherein, in order to generate key frame representation that can accurately describe the video content, a multi-granularity representation method is adopted to ensure the comprehensiveness and accuracy of the description. The title text is a coarse-grained text, which is used to summarize the main content described in the image. These contents (captions) provide a high-level understanding of the entire scene, helping to quickly grasp the core information of each key frame; through the target detection and instance labeling model in the multi-modal large model, detailed elements (label text) in each key frame are identified and labeled, including: object, action, attribute and other dimensional labels, so as to realize the fine representation of each key frame. The description text for each key frame can be represented as: Caption (n) represents the title text of the nth key frame, Tag (n) represents the label text of the nth key frame.

[0074] In an optional manner, the step of determining the target pseudo-supervised pair corresponding to the sample power video includes:

[0075] Using the context dynamic aggregation method of the large language model, and combining the multiple atomic segments corresponding to each visual segment, the initial pseudo-supervised pair corresponding to the sample power video is obtained.

[0076] Wherein, the prompt word template of the context dynamic aggregation task facing the large language model is initialized. The template first defines the role of the large language model as a video analyst by giving <example and video>, and then defines the task as merging and summarizing the initial atomic segments of the video and the description of each segment, and generating a pseudo-supervised pair (visual segment and corresponding description) as the initial pseudo-supervised pair R init = LLM(V sb ); represents the atomic segment obtained by the foregoing visual boundary segmentation and semantic boundary segmentation of each visual segment corresponding to the sample power video.

[0077] It should be noted that the evaluation criteria of the large language model context dynamic aggregation result in the embodiment are as follows: ① non-redundancy: ensure that each segment describes a unique activity and integrates segments related to the same activity. ② singularity: each segment should only describe one activity, avoiding mixing multiple activities in a single segment. ③ integrity: adjust the segment according to the overall information of the video, unify the segment description, and emphasize the time sequence and logical relationship. ④ non-fragmentation: avoid short segments (less than 5 seconds) and ensure that each segment is not fragmented. ⑤ accuracy: based on the global information of the video, ensure that the description of each segment is accurate and there is no uncertainty.

[0078] The score of each initial pseudo-supervised pair corresponding to the any sample power video is obtained by using the large language model, the plurality of atomic segments corresponding to each visual segment in the any sample power video, and each initial pseudo-supervised pair corresponding to the any sample power video.

[0079] The plurality of atomic segments corresponding to each visual segment in the any sample power video and any initial pseudo-supervised pair corresponding to the any sample power video are input into the large language model to obtain the score of the any initial pseudo-supervised pair, and the process is repeated until the score of each initial pseudo-supervised pair corresponding to the any sample power video is obtained.

[0080] To ensure the accuracy of the large language model for dynamic aggregation of visual segments, the embodiment requires the large language model to score each initial pseudo-supervised pair R init based on more than 5 criteria (for example, accuracy, consistency, relevance, completeness, and interpretability, etc.), that is, S = LLM(V sb , R init ); S represents the score of the initial pseudo-supervised pair.

[0081] The initial pseudo-supervised pair with a score higher than the target threshold is determined as the target pseudo-supervised pair corresponding to the any sample power video.

[0082] The value of the target threshold t can be set according to the actual situation, which is not limited herein. The number of target pseudo-supervised pairs is at least one.

[0083] It should be noted that Figure 2 The overall principle schematic diagram of the embodiment is shown. The loss function of the cross-modal power video positioning model is as follows: L is the loss function of a single target supervised pair.

[0084] As shown in Table 1, the positioning accuracy and efficiency comparison results of the cross-modal power video positioning model trained by the method in the embodiment and the international leading similar model are obtained. As shown in Table 1, compared with the similar comparison model, the positioning accuracy of the cross-modal power video positioning model in the embodiment has significant superiority.

[0085] Table 1:

[0086]

[0087] The embodiment first performs visual boundary segmentation and semantic boundary segmentation. Specifically, the video is segmented into visual boundaries according to the change of visual features, and then key frames are extracted from each visual segment, and coarse-grained titles and fine-grained labels are generated as the description text of the key frames. The visual segment and the description text of the corresponding key frame are input into a large language model, and the large language model is used to further divide the semantic boundary according to the semantic change in the key frame representation, and the atomic segment of the video is obtained. The embodiment then generates pseudo-supervision based on context dynamic aggregation. Specifically, the atomic segments of the entire video and their description texts are input into a large language model, and the global information of the video is analyzed based on the global information generation of the video to perform context aggregation. By checking the temporal and logical relationship between the atomic segments, the LLM merges the segments with similar semantics, and summarizes and generates queries for each merged segment. Through this process, the context aggregation is realized, and the non-redundant final segment with new context-related queries is obtained, thereby generating high-quality pseudo-supervised pairs. At the same time, multiple instructions and standards are set in the prompt to guide the large language model to score itself better, so that the large language model can better understand the task. Finally, based on the generated pseudo-supervised pairs, the fully supervised cross-modal power video positioning model is trained, and robust and efficient zero-shot cross-modal video positioning is realized.

[0088] The technical scheme of the embodiment generates time-dynamic pseudo-supervised pairs with boundary-aware features through visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation, and trains a fully supervised cross-modal power video positioning model based on the generated pseudo-supervised pairs, thereby realizing robust and efficient zero-shot cross-modal power abnormal behavior positioning, and improving the accuracy and efficiency of cross-modal power video positioning.

[0089] Figure 3 An embodiment of a cross-modal power video positioning system 200 provided by the present application is shown in a structural schematic diagram. As shown in the figure, the system 200 includes a generation module 210, a training module 220 and a detection module 230. Figure 3

[0090] ​The generation module 210 is configured to sequentially perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video to obtain at least one target pseudo-supervision pair corresponding to each sample power video; each target pseudo-supervision pair includes: a corresponding sample power video, a pseudo query text and a start and end time of the pseudo query text in the sample power video.

[0091] The training module 220 is configured to train a preset cross-modal power video positioning model based on the target pseudo-supervision pair corresponding to each sample power video to obtain a trained cross-modal power video positioning model.

[0092] The detection module 230 is configured to input a to-be-detected power video and a target query text into the trained cross-modal power video positioning model to obtain a video positioning result of the to-be-detected power video.

[0093] In an optional manner, the generation module 210 is specifically configured to:

[0094] perform visual boundary segmentation on any sample power video to obtain a plurality of visual segments, and perform semantic boundary segmentation on each visual segment to obtain a plurality of atomic segments corresponding to each visual segment;

[0095] determine the target pseudo-supervision pair corresponding to the any sample power video by using a context dynamic aggregation manner and in combination with the plurality of atomic segments corresponding to each visual segment.

[0096] In an optional manner, the generation module 210 is specifically configured to:

[0097] extract at least one key frame in the any visual segment, and obtain a description text corresponding to each key frame in the any visual segment;

[0098] input the any visual segment and the description text corresponding to each key frame in the any visual segment into a large language model, and obtain a plurality of atomic segments corresponding to the any visual segment in combination with a prompt word.

[0099] In an optional manner, the description text includes title text and label text; and the generation module 210 is specifically configured to:

[0100] obtain the title text corresponding to each key frame in the any visual segment by using an image captioning model, and obtain the label text corresponding to each key frame in the any visual segment by using a multi-modal large model.

[0101] In an optional manner, the generation module 210 is specifically configured to:

[0102] The context of the large language model is dynamically aggregated, and combined with the plurality of atomic segments corresponding to each visual segment, to obtain each initial pseudo-supervision pair corresponding to the any sample power video;

[0103] The large language model, the plurality of atomic segments corresponding to each visual segment in the any sample power video, and each initial pseudo-supervision pair corresponding to the any sample power video are used to obtain a score of each initial pseudo-supervision pair corresponding to the any sample power video.

[0104] The initial pseudo-supervision pair with a score higher than a target threshold is determined as a target pseudo-supervision pair corresponding to the any sample power video.

[0105] It should be noted that the beneficial effects of the cross-modal power video positioning system provided by the above embodiments are the same as those of the cross-modal power video positioning method described above, and will not be repeated here. In addition, when the system provided by the above embodiments implements its functions, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the above described functions. In addition, the system and method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0106] Among them, the cross-modal power video positioning system of the application can be a computer program (including program code) running in a computer device, for example, the cross-modal power video positioning system of the application is an application software, which can be used to execute the corresponding steps in the cross-modal power video positioning method of the application.

[0107] In some embodiments, the cross-modal power video positioning system of the application can be realized in a combination of software and hardware, for example, the cross-modal power video positioning system of the application can be a hardware decoding processor form of processor, which is programmed to execute the cross-modal power video positioning method of the application, for example, the hardware decoding processor form of processor can use one or more application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic device (PLD, Programmable Logic Device), complex programmable logic device (CPLD, Complex Programmable Logic Device), field programmable gate array (FPGA, Field-Programmable Gate Array) or other electronic components.

[0108] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the names of the modules do not limit the modules themselves.

[0109] An electronic device according to an embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned cross-modal power video positioning methods is implemented. That is, an electronic device according to an embodiment of the present application can include, but is not limited to, a processor and a memory; the memory is configured to store a computer program; and the processor is configured to execute the cross-modal power video positioning method according to any of the embodiments of the present application by invoking the computer program.

[0110] In an optional embodiment, an electronic device is provided, as shown in Figure 4 Figure 4 The electronic device 4000 shown in the figure includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, through a bus 4002. Optionally, the electronic device 4000 can also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual application, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not limit the embodiments of the present application.

[0111] The processor 4001 can be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 4001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0112] ​The bus 4002 can include a path that transmits information between the above-described components. The bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 4002 can be divided into an address bus, a data bus, a control bus, or the like. For convenience of representation, Figure 4 The bus 4002 is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.

[0113] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0114] The memory 4003 is used to store application program code (computer program) for executing the scheme of the present application, and is controlled by the processor 4001 to execute. The processor 4001 is used to execute the application program code stored in the memory 4003 to realize the content shown in the foregoing method embodiments.

[0115] Among them, the electronic device can also be a terminal device, and the terminal device can be any terminal device that can install an application and access a webpage through the application, including at least one of a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart television, and a smart vehicle device.

[0116] It should be noted that, Figure 4 The electronic device shown is only an example and should not limit the functions and use range of the embodiments of the present application.

[0117] The computer readable storage medium of the embodiment of the present application, the computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement any of the above cross-modal power video positioning methods.

[0118] Optionally, the computer readable storage medium can be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0119] In the exemplary embodiments, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the electronic device execute the above cross-modal power video positioning method.

[0120] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++ or conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0121] It should be understood that the flow diagrams and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of various embodiments of the present application. In this regard, each block in the flow diagrams and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each of the blocks of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0122] The computer readable storage medium of embodiments of the present application can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, the computer readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0123] The computer readable storage medium described above can bear one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0124] The above description merely provides preferred embodiments of the present application and a principle of applied technology. It should be understood by those skilled in the art that the disclosed range of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the disclosed concept. For example, the above technical features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

[0125] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and do not represent a specific order or sequential order. The order of use of similar objects can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described.

[0126] Those skilled in the art know that the present application can be implemented as a system, a method or a computer program product, so the present application can be specifically implemented as follows: it can be a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present application can also be implemented as a computer program product in one or more computer readable media, which contains computer readable program code.

[0127] Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A cross-modal electric power video positioning method, characterized in that, The method comprises the following steps: visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation are sequentially performed on each sample power video to obtain at least one target pseudo-supervision pair corresponding to each sample power video; wherein each target pseudo-supervision pair comprises: a corresponding sample power video, a pseudo query text and a start and end time of the pseudo query text in the sample power video; based on the target pseudo-supervision pair corresponding to each sample power video, a preset cross-modal power video positioning model is trained to obtain a trained cross-modal power video positioning model; the to-be-tested power video and the target query text are input into the trained cross-modal power video positioning model to obtain a video positioning result of the to-be-tested power video; the steps of sequentially performing visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on any sample power video to obtain at least one target pseudo-supervision pair corresponding to the sample power video, comprising: visual boundary segmentation is performed on the any sample power video to obtain a plurality of visual segments, and semantic boundary segmentation is performed on each visual segment to obtain a plurality of atomic segments corresponding to each visual segment; determine the target pseudo-supervision pair corresponding to the any sample power video by using a context dynamic aggregation manner and combining the plurality of atomic segments corresponding to each visual segment; the steps of performing semantic boundary segmentation on any visual segment to obtain a plurality of atomic segments corresponding to the any visual segment, comprising: extract at least one key frame in the any visual segment, and obtain a description text corresponding to each key frame in the any visual segment; input the any visual segment and the description text corresponding to each key frame in the any visual segment into a large language model, and obtain a plurality of atomic segments corresponding to the any visual segment by combining a prompt word; the description text comprises title text and label text; the steps of obtaining the description text corresponding to each key frame in the any visual segment, comprising: obtain the title text corresponding to each key frame in the any visual segment by using an image captioning model, and obtain the label text corresponding to each key frame in the any visual segment by using a multi-modal large model; the steps of determining the target pseudo-supervision pair corresponding to the any sample power video by using a context dynamic aggregation manner and combining the plurality of atomic segments corresponding to each visual segment, comprising: obtain each initial pseudo-supervision pair corresponding to the any sample power video by using a context dynamic aggregation manner of a large language model and combining the plurality of atomic segments corresponding to each visual segment; obtain a score of each initial pseudo-supervision pair corresponding to the any sample power video by using a large language model, the plurality of atomic segments corresponding to each visual segment in the any sample power video and each initial pseudo-supervision pair corresponding to the any sample power video; determine the initial pseudo-supervision pair with a score higher than a target threshold as the target pseudo-supervision pair corresponding to the any sample power video.

2. A cross-modal electric power video positioning system, characterized by, The method comprises the following steps: a generation module, a training module and a detection module; The generation module is configured to sequentially perform visual boundary segmentation, semantic boundary segmentation and context dynamic aggregation on each sample power video to obtain at least one target pseudo-supervision pair corresponding to each sample power video; each target pseudo-supervision pair comprises a corresponding sample power video, a pseudo-query text and a start and end time of the pseudo-query text in the sample power video; The training module is configured to train the preset cross-modal power video positioning model based on the target pseudo-supervision pair corresponding to each sample power video to obtain a trained cross-modal power video positioning model; The detection module is configured to input a to-be-detected power video and a target query text into the trained cross-modal power video positioning model to obtain a video positioning result of the to-be-detected power video. The generation module is specifically configured to: perform visual boundary segmentation on any sample power video to obtain a plurality of visual segments, and perform semantic boundary segmentation on each visual segment to obtain a plurality of atomic segments corresponding to each visual segment; determine the target pseudo-supervision pair corresponding to the any sample power video by using a context dynamic aggregation manner and in combination with the plurality of atomic segments corresponding to each visual segment; The generation module is specifically configured to: extract at least one key frame in the any visual segment, and obtain a description text corresponding to each key frame in the any visual segment; input the any visual segment and the description text corresponding to each key frame in the any visual segment into a large language model, and obtain a plurality of atomic segments corresponding to the any visual segment in combination with a prompt word; The description text comprises title text and label text; and the generation module is specifically configured to: obtain the title text corresponding to each key frame in the any visual segment by using an image captioning model, and obtain the label text corresponding to each key frame in the any visual segment by using a multi-modal large model; The generation module is specifically configured to: obtain each initial pseudo-supervision pair corresponding to the any sample power video by using a context dynamic aggregation manner of a large language model and in combination with the plurality of atomic segments corresponding to each visual segment; obtain a score of each initial pseudo-supervision pair corresponding to the any sample power video by using the large language model, the plurality of atomic segments corresponding to each visual segment in the any sample power video and each initial pseudo-supervision pair corresponding to the any sample power video; determine an initial pseudo-supervision pair with a score higher than a target threshold as the target pseudo-supervision pair corresponding to the any sample power video.

3. An electronic device, comprising: The electronic device comprises a processor and a memory coupled with the processor, and the memory stores at least one computer program, which is loaded and executed by the processor, so that the electronic device implements the cross-modal power video positioning method of claim 1.

4. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, which is loaded and executed by the processor, so that the computer readable storage medium implements the cross-modal power video positioning method of claim 1.

Images