Efficient and accurate retrieval method and device for music variety multi-camera video materials

Abstract

The invention discloses an efficient and accurate retrieval method and device for music variety multi-camera video materials, and belongs to the field of cross-modal retrieval, and the method comprises the steps: obtaining a multi-camera material block Snippet; the initial positions of all the multi-camera material blocks Snippet blocks are determined; generating a music variety editing script by using the large model; searching a multi-camera video material according to the editing script generated in the step S3; and according to the retrieved block Snippet, carrying out splicing by considering a timeline sequence. According to the method, the retrieval accuracy and the retrieval accuracy efficiency of the multi-camera video materials are improved.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal retrieval, and more specifically, to an efficient and accurate retrieval method and device for multi-camera video materials of music variety shows. Background Art

[0002] With the rapid development of digital media technology, the production and dissemination of music variety shows are increasing, bringing challenges to the management and retrieval of massive video materials. Traditional video retrieval methods rely on text tags or simple keyword matching, and often seem powerless when faced with complex and changeable music variety show scenes. When processing such video materials, existing technologies often ignore the correlation and complementarity between different camera positions, resulting in low accuracy and efficiency of retrieval results. In addition, most methods focus on single-modal information retrieval, such as based only on images or only on audio, while music variety shows usually contain rich visual and auditory elements, and single-modal information is difficult to fully capture the essence of the show. Therefore, it is particularly important to develop an efficient and accurate retrieval method for multi-camera video materials of music variety shows. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method and device for efficiently and accurately searching multi-camera video materials of music variety shows, thereby improving the search accuracy and efficiency.

[0004] The object of the present invention is achieved through the following solutions:

[0005] An efficient and accurate retrieval method for multi-camera video materials of music variety shows, comprising the following steps:

[0006] S1: Get the multi-camera material block Snippet;

[0007] S2: Determine the starting position of all multi-camera material blocks Snippet blocks;

[0008] S3: Generate music variety show editing scripts using the big model;

[0009] S4: Retrieving multi-camera video materials according to the editing script generated in step S3;

[0010] S5: Based on the retrieved snippets, the timeline order is considered and spliced.

[0011] Furthermore, in step S1, obtaining the multi-camera material block Snippet specifically includes the following sub-steps:

[0012] S101: Obtain video materials from all camera positions of a music variety show and number them according to the camera positions;

[0013] S102: For each video material, separate and obtain corresponding audio;

[0014] S103: Obtaining an audio pause point based on audio analysis;

[0015] S104: Analyzing the audio using a sound event detection algorithm to obtain a time point at which an audio event occurs, where the time point at which the audio event occurs includes a drum beat and an applause beat;

[0016] S105: According to the audio pause points obtained in step S103 and the drum beats and applause points obtained in step S104, the corresponding video material is divided into a plurality of snippets;

[0017] S106: Analyze the snippets of each camera position and decide whether to merge or split them according to their duration.

[0018] Further, in step S103, obtaining the audio pause point based on the audio analysis specifically includes the following sub-steps:

[0019] First, the timeline audio energy is obtained according to the audio timeline, and the timeline starting points where the timeline audio energy is 0 or close to 0 are found; and the average midpoint of these timeline starting points is used as the audio pause point.

[0020] Furthermore, in step S106, the snippets of each camera position are analyzed and merged or split according to the duration, which specifically includes the following sub-steps:

[0021] If the number of adjacent block snippets on the timeline is less than the set value of one, the two adjacent block snippets are merged into one block snippet; if the number of a single block snippet exceeds the set value of two, the block snippet is split into two block snippets according to the midpoint.

[0022] Furthermore, in step S2, determining the starting position of all the multi-camera material blocks Snippet blocks specifically includes the following sub-steps:

[0023] S201: extract frames from the snippets of each camera position, and separate the corresponding audio from each video material;

[0024] S202: for the frame extraction results of all camera positions, use the target detection algorithm to detect the frames in which the logo image of the music variety show appears in each camera position;

[0025] S203: using a sound event detection algorithm to detect the time point at which the opening music of the music variety show appears in each camera position for the audio of all cameras;

[0026] S204: For all camera positions, compare the earliest time point when the logo image appears and the earliest time point when the opening music appears, then take the smaller time point as the starting point of each camera position, and then delete the block snippet before the starting point; if a camera position finds neither the time point when the logo of the music variety show appears nor the time point when the opening music of the music variety show appears, the video material of this camera position will not be processed, and all of them will be included in the subsequent search range.

[0027] Furthermore, in step S3, the large model is used to generate a music variety show editing script, specifically by writing prompts to generate the music variety show editing script.

[0028] Furthermore, in step S4, the retrieval of multi-camera video materials specifically includes the following sub-steps:

[0029] S401: extracting frames from the snippets of each camera position, and dividing each frame image into frame blocks;

[0030] S402: For each complete frame, use the spatial encoder to perform embedded coding, denoted as Where f is the encoder mapping function, N is the camera number, and m is the frame number of the camera;

[0031] S403: For each frame block in step S401, use the same spatial encoder in step S402 to perform embedded coding, denoted as Where f is the encoder mapping function, N is the camera number, m is the frame number of the camera, and pk is the frame block number of the frame;

[0032] S404: For the music variety show editing script output in step S3, embedding encoding is performed on each shot using a text encoder, denoted as f s ; Where f is the encoder mapping function and s is the shot text number;

[0033] S405: For the text embedding coding of step S404 and the image space coding of the complete frame of step S402, a parameter-free gating unit is used to complete the interaction between the text embedding coding and the image space coding of the complete frame, and a recall result set is obtained:

[0034]

[0035] Among them, v N recall Represents the recall result set of the Nth seat. All subsequent recall operations are completed in this recall result set. The vector result representing the interaction between the S-th shot text and the M-th complete frame on N camera positions after passing through a parameter-free gating unit;

[0036] During the training phase, The calculation formula is as follows:

[0037]

[0038] In the inference phase, The calculation formula is as follows:

[0039]

[0040] in, represents the text embedding code of the S-th shot script, represents the image space embedding coding of the Mth frame, π represents the temperature coefficient, and its value is between 0 and 1;

[0041] S406: Recall vector set v for the Nth camera position N recall , for each shot text encoding, take a subset Take the first K cameras in the sorting order. The camera position has recalled SK frames in total. All N cameras have recalled NSK frames in total. The frame numbers of these recalled cameras are recorded for screening in step S407.

[0042] S407: The text embedding coding of step S404 and the image space coding of the frame block of step S403 are passed through a parameter-free gating unit, thereby completing the interaction between the text embedding coding and the image space coding of the frame block, and obtaining a sorting vector set:

[0043]

[0044] Among them, v N sort Represents the sorted result set of the Nth seat; The vector result representing the interaction between the S-th shot text and all p frame blocks of the M-th frame on the N-camera after passing through a parameter-free gating unit;

[0045] In the training phase The calculation formula is as follows:

[0046]

[0047] In the inference stage The calculation formula is as follows:

[0048]

[0049] Among them, f tS represents the text embedding code of the S-th shot script, f mprepresents the image space embedding coding of the p-th frame block of the M-th frame image, π represents the temperature coefficient, and its value is between 0 and 1; for this sorting result set, according to the frame numbers recorded in step S406, only the sorting result sets of these frame numbers are calculated, and then each script ts selects the first-ranked result as the final sorting selection result; if the first-ranked result selected by a previous script is the same as that selected by a later script, the later script selects from the second-ranked result of the script, and so on; the frame number selected by each script is recorded;

[0050] S408: The frame number selected in step S407 is matched to the block Snippet number, and each script selects the corresponding block Snippet as the final search result.

[0051] Furthermore, in step S5, the retrieved snippets are spliced ​​in consideration of the timeline order, specifically including: performing secondary arrangement according to the snippet selected in step S408 and the event sequence of the snippet to determine a snippet sequence that complies with the time sequence.

[0052] Further, in step S202, the frame extraction results for all camera positions use a target detection algorithm to detect frames in which the logo image of the music variety show appears at each camera position, specifically including the sub-steps of: if a camera position detects multiple frames containing the logo image of the music variety show, then taking the time point of the earliest appearing frame;

[0053] In step S203, the audio of all camera positions uses a sound event detection algorithm to detect the time point when the opening music of the music variety show appears at each camera position, which specifically includes the following sub-steps: if a camera position detects multiple opening music containing the music variety show, the time point when the opening music appears earliest is taken.

[0054] A highly efficient and accurate retrieval device for multi-camera video materials of music variety shows, comprising a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, any of the methods described above is executed.

[0055] The beneficial effects of the present invention include:

[0056] (1) The present invention takes into account the coarse and fine granularity of different image sizes and proposes a new cross-modal retrieval method, which includes obtaining multi-camera material block Snippets, determining the starting positions of all multi-camera material block Snippet blocks, using a large model to generate a music variety show editing script, retrieving multi-camera video materials according to the generated editing script, and splicing the retrieved multi-camera video materials according to the retrieved block Snippets, etc. Through the above steps, the cross-modal retrieval model's overall and local perception capabilities are improved, and the retrieval accuracy is improved.

[0057] (2) The method of the present invention improves the retrieval efficiency of a large amount of multi-camera materials by greatly improving the retrieval efficiency of the cross-modal retrieval model. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 Flowchart for cross-modal retrieval;

[0060] Figure 2 The figure is a flowchart of the steps of the method according to the embodiment of the present invention. DETAILED DESCRIPTION

[0061] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0062] In view of the current situation in the background, the inventors of this application believe that: cross-modal retrieval technology improves retrieval accuracy and efficiency by integrating information from different modalities, but faces the challenges of the complexity of multi-modal information fusion, the diversity of multi-camera video materials, and the balance between retrieval efficiency and accuracy in the retrieval of music variety show video materials. In response to these challenges, the present invention specifically proposes an innovative method for retrieval of multi-camera video materials for music variety shows, aiming to address the limitations of the existing technology and improve retrieval efficiency and accuracy. Furthermore, the technical solution of the present invention achieves efficient and accurate retrieval of music variety show video materials in a preferred embodiment through the innovation of cross-modal retrieval models and retrieval strategies, such as Figure 1 and Figure 2 As shown, the specific implementation process includes the following steps:

[0063] Step S1: Obtain multi-camera material Snippet (block).

[0064] Step S2: Determine the starting position of all multi-camera material snippets (blocks).

[0065] Step S3: The large model generates a music variety show editing script.

[0066] Step S4: Retrieve multi-camera video materials.

[0067] Step S5: splicing the search results into a timeline.

[0068] In other embodiments of the present invention, in step S1 of the above scheme, obtaining the multi-camera video material Snippet (block) means: analyzing the video materials of all cameras, identifying the special audio rhythm points in the video materials and then dividing the video materials into Snippets (blocks). The specific implementation steps of step S1 are as follows:

[0069] Step S101: Obtain video materials from all camera positions of a music variety show and number them according to the camera positions.

[0070] Step S102: For each video material, separate and obtain the corresponding audio.

[0071] Step S103: Based on the audio, analyze and obtain the audio pause point. First, obtain the timeline audio energy according to the audio timeline, and find the timeline starting point where the timeline audio energy is 0 or close to 0. The average midpoint of these timeline starting points is used as the audio pause point.

[0072] Step S104: Based on the audio, use a sound event detection algorithm to analyze and obtain the time points at which audio events such as drum beats and applause points occur.

[0073] Step S105: According to the audio pause points obtained in step S103 and the drum beats and applause points obtained in step S104, the corresponding video material is divided into multiple snippets (blocks).

[0074] Step S106: Analyze the snippets of each camera position and decide whether to merge or split them according to the duration. If the adjacent snippets on the timeline are too short (less than 5 seconds), merge the two adjacent snippets into one snippet; if a single snippet is too long (more than 60 seconds), split the snippet into two snippets at the midpoint.

[0075] In step S2 of the above scheme, determining the starting position of all multi-camera material snippets (blocks) means: determining the time point when the music variety show starts, thereby deleting the snippets (blocks) before the start time point and not including them in subsequent retrieval. The specific implementation steps of step S2 are as follows:

[0076] Step S201: extract frames from the Snippet blocks of each camera position (extract frames every 5 frames); at the same time, separate the corresponding audio for each video material.

[0077] Step S202: Based on the frame extraction results of all camera positions, use the target detection algorithm to detect the frames in which the logo image of the music variety show appears in each camera position. When a camera position detects multiple frames containing the logo image of the music variety show, the time point of the earliest frame is taken.

[0078] Step S203: For the audio of all camera positions, use the sound event detection algorithm to detect the time point when the opening music of the music variety show appears at each camera position. When a camera position detects multiple opening music containing the music variety show, the time point when the opening music appears earliest is selected.

[0079] Step S204: For all camera positions, compare the earliest time point of the logo image and the earliest time point of the opening music, then take the smaller time point as the starting point of each camera position, and then delete the snippet before the starting point. If a camera position does not find the time point of the logo of the music variety show, nor the time point of the opening music of the music variety show, the video material of the camera position will not be processed, and all will be included in the subsequent search range.

[0080] In step S3 of the above scheme, generating a music variety show editing script based on a large model means: using the prompt engineering method to write a suitable prompt to generate a music variety show editing script. The specific implementation steps of step S3 are as follows:

[0081] Step S301: Write a prompt. The specific prompt is as follows:

[0082] **Mission Statement**:

[0083] You are a professional editor of a music variety show. You know the host, guest singers, songs and other information of this music show. You need to generate an editing script based on this known information and taking into account the following editing requirements. The script should include the scene type (long shot, full shot, medium shot, close shot, close-up), angle (front, side, back), object (music variety show logo, sponsor logo), and people appearing on the screen:

[0084] Host Information:

[0085] {Host Name}

[0086] Singers and song information:

[0087] {Guest name: song title}

[0088] **Editing Request**:

[0089] Music variety shows need a relatively grand scene at the beginning, and then the host will appear;

[0090] The host will announce each guest singer before they begin singing;

[0091] The difference in the shot size between consecutive shots should not be too great (for example, a long shot cannot be followed immediately by a close-up shot), and the gradual change in shot size should be considered;

[0092] The host's shots are mainly frontal (accounting for 80%), supplemented by side shots (accounting for 20%); except for the opening, the host's shots are as few as possible, such as long shots and panoramic shots;

[0093] The requirements for the shots of the singing guests are varied. The front shot of the singing guest just entering the venue can be a long shot. The shots during the singing process are usually medium shots, close shots, and close-ups. The angles are mainly front (accounting for 40%) and side (accounting for 40%). The back (i.e. the scene of interaction with the audience) (accounting for 20%) should also be considered.

[0094] All singing guests generate at least 10 shots;

[0095] Objects should appropriately display the logo of the music variety show and the logo of the sponsor;

[0096] When no one appears, it must be an empty mirror, and when someone appears, it must be a non-empty mirror;

[0097] Each shot is output in json format {"scene":"xxx","angle":"xxx","object":"xxx","person in shot":"xxx","is there an empty shot":"xxx"};

[0098] **Output**.

[0099] In step S4 of the above scheme, retrieving multi-camera video materials means: according to the editing script generated in step S3, using the "cross-modal retrieval method" of the present invention, efficiently and accurately retrieve multi-camera video materials. The specific implementation steps of step S4 are as follows:

[0100] Step S401: extract frames from the Snippet blocks of each camera position (extract frames every 5 frames), and divide each frame image into 3*3 frame blocks (that is, each frame is divided into 9 blocks of the same size);

[0101] Step S402: For each complete frame, use Spatial Encoder to perform embedded coding, denoted as Where f is the encoder mapping function, N is the camera number, and m is the frame number of the camera;

[0102] Step S403: For each frame block in step S401, use the same Spatial Encoder in step S402 for embedded coding, denoted as Where f is the encoder mapping function, N is the camera number, m is the frame number of the camera, and pk is the frame block number of the frame;

[0103] Step S404: For the music variety show editing script output in step S3, use TextEncoder to embed code each shot, denoted as f s . Where f is the encoder mapping function and s is the shot text number.

[0104] Step S405: The text embedding coding of step S404 and the image space coding of the complete frame of step S402 are passed through a parameter-free gating unit, thereby completing the interaction between the text embedding coding and the image space coding of the complete frame, and obtaining a recall result set:

[0105]

[0106] Among them, v N recall Represents the recall result set of the Nth camera position. All subsequent recall operations are completed in this recall result set. The vector result representing the interaction between the S-th shot text and the M-th complete frame on N camera positions after passing through a parameter-free gating unit.

[0107] During the training phase, The calculation formula of (i.e., no parameter gating unit) is as follows:

[0108]

[0109] In the inference phase, The calculation formula of (i.e., no parameter gating unit) is as follows:

[0110]

[0111] in, represents the text embedding code of the S-th shot script, It represents the image space embedding coding of the Mth frame, and π represents the temperature coefficient, which takes a value between 0 and 1.

[0112] Step S406: For the recall vector set v of the Nth camera position N recall , for each shot text encoding, take a subset Take TOP K. Therefore, the camera position has recalled SK frames in total, and all N camera positions have recalled NSK frames in total. The frame numbers of these recalled camera positions are recorded for screening in step S407.

[0113] Step S407: The text embedding coding of step S404 and the image space coding of the frame block of step S403 are passed through a parameter-free gating unit, thereby completing the interaction between the text embedding coding and the image space coding of the frame block, and obtaining a sorting vector set:

[0114]

[0115] Among them, v N sort Represents the sorted result set of the Nth seat. The vector result representing the interaction between the S-th shot text and all p frame blocks of the M-th frame on the N-camera after passing through a parameter-free gating unit.

[0116] In the training phase The calculation formula of (i.e., no parameter gating unit) is as follows:

[0117]

[0118] In the inference stage The calculation formula of (i.e., no parameter gating unit) is as follows:

[0119]

[0120] in, represents the text embedding code of the S-th shot script, f mp It represents the image space embedding coding of the p-th frame block of the M-th frame image, and π represents the temperature coefficient, which takes a value between 0 and 1.

[0121] It should be noted that for this sorting result set, it is first necessary to calculate only the sorting result set of these frame numbers according to the frame numbers recorded in step S406, so as to reduce the amount of sorting calculation. Finally, each script ts selects the top1 as the final sorting selection result. If the top1 selected by the previous script is the same as that of the following script, the following script will be selected from the top2 sorting result of the script, and so on. Record the frame number selected by each script.

[0122] Step S408: correspond the frame number selected in step S407 to the Snippet number, and select the corresponding Snippet for each script as the final search result.

[0123] Through experimental comparison, the multimodal retrieval method of recalling first and then sorting improves the retrieval efficiency by 48% compared with the method of directly adopting step S407 (that is, no recall operation is performed before); the retrieval effect is improved by 41% compared with the method of directly adopting step S405 (that is, directly taking the recall result as the final retrieval result), thereby achieving a balance between multimodal retrieval effect and efficiency and achieving the purpose of efficient and accurate retrieval.

[0124] In step S5 of the above solution, the timeline splicing of the search results refers to: splicing the retrieved Snippets in consideration of the timeline sequence. The specific implementation steps of step S5 are as follows:

[0125] Step S501: performing secondary arrangement according to the Snippets selected in step S408 and the event sequence of the Snippets to determine a Snippet sequence that complies with the time sequence.

[0126] The cross-modal retrieval model of the embodiment of the present invention simultaneously considers the coarse and fine granularity (such as frames, frame blocks) information of different image sizes, improves the overall and local perception capabilities of the cross-modal retrieval model, and thus improves the retrieval accuracy.

[0127] The embodiment of the present invention adopts the idea of ​​recalling first and then re-ranking, which greatly improves the retrieval efficiency of the cross-modal retrieval model, thereby improving the retrieval efficiency of a large amount of multi-camera materials.

[0128] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0129] According to one aspect of an embodiment of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations.

[0130] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

Claims

1. An efficient and accurate retrieval method for multi-camera video materials of music variety shows, characterized in that: The following steps are involved: S1: Get the multi-camera material block Snippet; S2: Determine the starting position of all multi-camera material blocks Snippet blocks; S3: Generate music variety show editing scripts using the big model; S4: Retrieving multi-camera video materials according to the editing script generated in step S3; S5: Based on the retrieved snippets, the timeline order is considered and spliced.

2. The efficient and accurate retrieval method for multi-camera video materials of music variety shows according to claim 1 is characterized in that: In step S1, the step of obtaining a multi-camera material block Snippet specifically includes the following sub-steps: S101: Obtain video materials from all camera positions of a music variety show and number them according to the camera positions; S102: For each video material, separate and obtain corresponding audio; S103: Obtaining an audio pause point based on audio analysis; S104: Analyzing the audio using a sound event detection algorithm to obtain a time point at which an audio event occurs, where the time point at which the audio event occurs includes a drum beat and an applause beat; S105: According to the audio pause points obtained in step S103 and the drum beats and applause points obtained in step S104, the corresponding video material is divided into a plurality of snippets; S106: Analyze the snippets of each camera position and decide whether to merge or split them according to their duration.

3. The efficient and accurate retrieval method for multi-camera video materials of music variety shows according to claim 2 is characterized in that: In step S103, obtaining the audio pause point based on the audio analysis specifically includes the following sub-steps: First, the timeline audio energy is obtained according to the audio timeline, and the timeline starting points where the timeline audio energy is 0 or close to 0 are found; and the average midpoint of these timeline starting points is used as the audio pause point.

4. The efficient and accurate retrieval method for multi-camera video materials of music variety shows according to claim 2 is characterized in that: In step S106, the snippets of each camera position are analyzed and merged or split according to the duration, which specifically includes the following sub-steps: If the number of adjacent block snippets on the timeline is less than the set value of one, the two adjacent block snippets are merged into one block snippet; if the number of a single block snippet exceeds the set value of two, the block snippet is split into two block snippets according to the midpoint.

5. The efficient and accurate retrieval method for multi-camera video materials of music variety shows according to claim 1 is characterized in that: In step S2, the starting position of all the multi-camera material blocks Snippet blocks is determined, which specifically includes the following sub-steps: S201: extract frames from the snippets of each camera position, and separate the corresponding audio from each video material; S202: for the frame extraction results of all camera positions, use the target detection algorithm to detect the frames in which the logo image of the music variety show appears in each camera position; S203: using a sound event detection algorithm to detect the time point at which the opening music of the music variety show appears in each camera position for the audio of all cameras; S204: For all camera positions, compare the earliest time point when the logo image appears and the earliest time point when the opening music appears, then take the smaller time point as the starting point of each camera position, and then delete the block snippet before the starting point; if a camera position finds neither the time point when the logo of the music variety show appears nor the time point when the opening music of the music variety show appears, the video material of this camera position will not be processed, and all of them will be included in the subsequent search range.

6. The efficient and accurate retrieval method for multi-camera video materials of music variety shows according to claim 1 is characterized in that: In step S3, the large model is used to generate a music variety show editing script, specifically by writing prompts to generate the music variety show editing script.

7. The efficient and accurate retrieval method for multi-camera video materials of music variety shows according to claim 1 is characterized in that: In step S4, the retrieval of multi-camera video materials specifically includes the following sub-steps: S401: extracting frames from the snippets of each camera position, and dividing each frame image into frame blocks; S402: For each complete frame, use the spatial encoder to perform embedded coding, denoted as Where f is the encoder mapping function, N is the camera number, and m is the frame number of the camera; S403: For each frame block in step S401, use the same spatial encoder in step S402 to perform embedded coding, denoted as Where f is the encoder mapping function, N is the camera number, m is the frame number of the camera, and pk is the frame block number of the frame; S404: For the music variety show editing script output in step S3, embedding encoding is performed on each shot using a text encoder, denoted as f s ; Where f is the encoder mapping function and s is the shot text number; S405: For the text embedding coding of step S404 and the image space coding of the complete frame of step S402, a parameter-free gating unit is used to complete the interaction between the text embedding coding and the image space coding of the complete frame, and a recall result set is obtained: Among them, v N recall Represents the recall result set of the Nth seat. All subsequent recall operations are completed in this recall result set. The vector result representing the interaction between the S-th shot text and the M-th complete frame on N camera positions after passing through a parameter-free gating unit; During the training phase, The calculation formula is as follows: In the inference phase, The calculation formula is as follows: Among them, f tS represents the text embedding code of the S-th shot script, represents the image space embedding coding of the Mth frame, π represents the temperature coefficient, and its value is between 0 and 1; S406: Recall vector set v for the Nth camera position N recall , for each shot text encoding, take a subset Take the first K cameras in the sorting order. The camera position has recalled SK frames in total. All N cameras have recalled NSK frames in total. The frame numbers of these recalled cameras are recorded for screening in step S407. S407: The text embedding coding of step S404 and the image space coding of the frame block of step S403 are passed through a parameter-free gating unit, thereby completing the interaction between the text embedding coding and the image space coding of the frame block, and obtaining a sorting vector set: Among them, v N sort Represents the sorted result set of the Nth seat; The vector result representing the interaction between the S-th shot text and all p frame blocks of the M-th frame on the N-camera after passing through a parameter-free gating unit; In the training phase The calculation formula is as follows: In the inference stage The calculation formula is as follows: Among them, f tS represents the text embedding code of the S-th shot script, f mp represents the image space embedding coding of the p-th frame block of the M-th frame image, π represents the temperature coefficient, and its value is between 0 and 1; for this sorting result set, according to the frame numbers recorded in step S406, only the sorting result sets of these frame numbers are calculated, and then each script ts selects the first-ranked result as the final sorting selection result; if the first-ranked result selected by a previous script is the same as that selected by a later script, the later script selects from the second-ranked result of the script, and so on; the frame number selected by each script is recorded; S408: The frame number selected in step S407 is matched to the block Snippet number, and each script selects the corresponding block Snippet as the final search result.

8. The efficient and accurate retrieval method for multi-camera video materials of music variety shows according to claim 1 is characterized in that: In step S5, the retrieved snippets are spliced ​​in consideration of the timeline order, specifically including: performing secondary arrangement according to the snippet selected in step S408 and the event sequence of the snippets to determine a snippet sequence that complies with the time sequence.

9. The efficient and accurate retrieval method for multi-camera video materials of music variety shows according to claim 5 is characterized in that: In step S202, the frame extraction results for all camera positions are used to detect frames in which the logo image of the music variety show appears at each camera position using a target detection algorithm, specifically including the following sub-steps: if a camera position detects multiple frames containing the logo image of the music variety show, then the time point of the earliest appearing frame is taken; In step S203, the audio of all camera positions uses a sound event detection algorithm to detect the time point when the opening music of the music variety show appears at each camera position, which specifically includes the following sub-steps: if a camera position detects multiple opening music containing the music variety show, the time point when the opening music appears earliest is taken.

10. An efficient and accurate retrieval device for multi-camera video materials of music variety shows, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method according to any one of claims 1 to 9 is executed.

Images