Video material screening method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video material screening method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for financial science and technology scenes and medical science and technology scenes. The method comprises the following steps: acquiring an original video material; performing video frame extraction on the original video material to obtain an initial video frame sequence; performing subtitle detection on the initial video frame sequence to obtain subtitle detection data; performing semantic extraction on the original video materials to obtain material semantic features; performing index construction based on the subtitle detection data, the material semantic features and the original video materials to obtain a video material database; obtaining a material screening request, and performing statement conversion on the material screening request to obtain a material query statement; and performing material screening on the video material database based on the material query statement to obtain a target video material. According to the embodiment of the invention, the video material screening efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and is applicable to financial technology scenarios and medical technology scenarios, and in particular to a method and device for screening video materials, an electronic device, and a storage medium. Background Art

[0002] Video assets can be used for video production, creation, or editing. They include a variety of original imagery, including live action footage and animated clips. They can be applied across multiple scenarios. For example, in fintech, suitable assets can be selected from financial video assets to create promotional and marketing videos for insurance and financial products. In healthcare, suitable assets can be selected from medical video assets to create promotional and educational videos.

[0003] At present, video material screening mainly adopts manual screening or tag-based methods to select relevant video materials from massive video materials. However, in actual use, due to manual screening or tag-based methods, it is inefficient to select video materials that meet the needs from massive materials.

[0004] Therefore, how to improve the efficiency of video material screening has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to propose a video material screening method and device, electronic equipment and storage medium, aiming to improve the efficiency of video material screening.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a method for screening video materials, the method comprising:

[0007] Get the original video material;

[0008] Extracting video frames from the original video material to obtain an initial video frame sequence;

[0009] Performing subtitle detection on the initial video frame sequence to obtain subtitle detection data;

[0010] Performing semantic extraction on the original video material to obtain semantic features of the material;

[0011] An index is constructed based on the subtitle detection data, the material semantic features and the original video material to obtain a video material database;

[0012] Obtaining a material screening request, and performing sentence conversion on the material screening request to obtain a material query sentence;

[0013] The video material database is screened based on the material query statement to obtain target video material.

[0014] In some embodiments, performing subtitle detection on the initial video frame sequence to obtain subtitle detection data includes:

[0015] Extracting visual features from the initial video frame sequence to obtain original video visual features;

[0016] Performing text feature extraction on the initial video frame sequence to obtain original subtitle text features;

[0017] Subtitle detection is performed based on the original video visual features and the original subtitle text features to obtain the subtitle detection data.

[0018] In some embodiments, performing subtitle detection based on the original video visual features and the original subtitle text features to obtain the subtitle detection data includes:

[0019] Performing feature alignment on the original video visual features and the original subtitle text features to obtain an initial alignment score;

[0020] Performing aggregation calculation on the initial alignment scores based on a preset sliding window to obtain an image-caption alignment score;

[0021] Subtitle recognition is performed based on the image-subtitle alignment score to obtain the subtitle detection data.

[0022] In some embodiments, aligning the original video visual features with the original subtitle text features to obtain an initial alignment score includes:

[0023] Obtaining preset difference adjustment parameters;

[0024] Calculating similarity between the original video visual features and the original subtitle text features to obtain feature similarity data;

[0025] Performing difference adjustment on the feature similarity data based on the difference adjustment parameter to obtain an alignment difference feature;

[0026] An activation calculation is performed on the alignment difference feature to obtain the initial alignment score.

[0027] In some embodiments, performing subtitle recognition based on the image-subtitle alignment score to obtain the subtitle detection data includes:

[0028] Performing subtitle classification based on the image subtitle alignment score to obtain a video frame subtitle type; wherein the video frame subtitle type is used to indicate whether the initial video frame sequence contains subtitles or does not contain subtitles;

[0029] Performing a status identification on the initial video frame sequence based on the video frame subtitle type to obtain a subtitle identification status;

[0030] The initial video frame sequence is continuously time-parsed based on the subtitle identification state to obtain the subtitle detection data.

[0031] In some embodiments, extracting text features from the initial video frame sequence to obtain original subtitle text features includes:

[0032] Performing text region recognition on the initial video frame sequence to obtain text region position information;

[0033] Performing character recognition on the initial video frame sequence based on the text region position information to obtain original video subtitle data;

[0034] Performing word segmentation processing on the original video subtitle data to obtain a subtitle word sequence;

[0035] Feature encoding is performed on the subtitle word sequence to obtain the original subtitle text features.

[0036] In some embodiments, extracting video frames from the original video material to obtain an initial video frame sequence includes:

[0037] Performing bitstream analysis on the original video material to obtain video bitstream data;

[0038] Performing entropy decoding on the video bitstream data to obtain video coding features;

[0039] Dequantizing the video coding features to obtain video pixel values;

[0040] Performing inverse transformation on the video pixel values ​​to obtain an original video frame sequence;

[0041] The original video frame sequence is grouped into frames to obtain the initial video frame sequence.

[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video material screening device, the device comprising:

[0043] Video material acquisition module, used to obtain original video material;

[0044] A video frame extraction module, configured to extract video frames from the original video material to obtain an initial video frame sequence;

[0045] A subtitle detection module, configured to perform subtitle detection on the initial video frame sequence to obtain subtitle detection data;

[0046] A semantic extraction module, configured to perform semantic extraction on the original video material to obtain semantic features of the material;

[0047] An index construction module, configured to construct an index based on the subtitle detection data, the material semantic features and the original video material to obtain a video material database;

[0048] A query statement conversion module is used to obtain a material screening request and perform statement conversion on the material screening request to obtain a material query statement;

[0049] The target material screening module is used to screen the video material database based on the material query statement to obtain the target video material.

[0050] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0051] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0052] The video material screening method and device, electronic device, and storage medium proposed in this application extract video frames from the acquired original video material to obtain an initial video frame sequence, facilitating subsequent subtitle detection and semantic extraction of the video content of the original video material. Next, subtitle detection is performed on the initial video frame sequence to accurately locate subtitle information and obtain subtitle detection data. Simultaneously, semantic extraction is performed on the original video material to explore the material's deep semantic features and obtain material semantic features. Furthermore, an index is constructed based on the subtitle detection data, material semantic features, and the original video material to obtain a video material database, which provides efficient retrieval capabilities. Finally, a material screening request is obtained, the material screening request is converted into a material query statement, and material screening is performed on the video material database based on the material query statement. This allows for rapid and accurate screening of target video material from a vast amount of material, saving manual search time and improving the efficiency and accuracy of video material screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the video material screening method provided in an embodiment of the present application;

[0054] Figure 2 yes Figure 1 Flowchart of step S102 in FIG.

[0055] Figure 3 yes Figure 1 Flowchart of step S103 in FIG.

[0056] Figure 4 yes Figure 3 Flowchart of step S302 in FIG.

[0057] Figure 5 yes Figure 3 Flowchart of step S303 in FIG.

[0058] Figure 6 yes Figure 5 Flowchart of step S501 in FIG.

[0059] Figure 7 yes Figure 5 Flowchart of step S503 in FIG.

[0060] Figure 8 Schematic diagram of the structure of the video material screening device provided in an embodiment of the present application;

[0061] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0065] First, let’s analyze some of the terms used in this application:

[0066] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0067] Video assets can be used for video production, creation, and editing. These assets include a variety of original footage, including live action footage and animated clips. Screening video assets ensures the content is relevant to the theme, high quality, and consistent in style, enhancing the overall impact of the video. Video assets can be applied across multiple scenarios. For example, in fintech, suitable assets can be selected from financial video assets to create promotional and marketing videos for insurance and wealth management products. In medical technology, suitable assets can be selected from medical video assets to create promotional and educational videos.

[0068] Video decoding is the process of restoring compressed video data to its original signal. During video transmission and storage, raw video is compressed to save bandwidth and storage space. During decoding, the decoder reverses the compressed data based on the encoding algorithm, performing steps such as analyzing the frame structure, restoring the color space, and reconstructing motion vectors. Ultimately, the decoder generates a continuous video image that can be played on a display device.

[0069] Subtitle detection refers to the technical means used to identify subtitle areas in videos or images. It is widely used in scenarios such as video retrieval, content review, and auxiliary generation. Typically, based on the differences in color, edges, and texture between subtitles and background, image processing algorithms such as edge detection, connected domain analysis, and texture classification are used to first extract potential subtitle areas. Then, the subtitles are screened and verified based on their spatiotemporal characteristics such as position, size, and appearance time. Ultimately, the subtitle location is determined, providing a foundation for subsequent subtitle extraction and recognition.

[0070] Semantic extraction involves extracting features that convey deep semantic information from data such as text, images, or videos using technologies like natural language processing and computer vision. Semantic extraction from videos can identify objects, scenes, actions, events, and their relationships within the video, converting them into structured semantic descriptions. This helps computers understand the video content and provides semantic support for subsequent retrieval, classification, recommendation, and analysis.

[0071] Currently, video material screening primarily relies on manual or tag-based methods to select relevant video material from massive amounts of video material. For example, this involves finding pure video material without subtitles, or identifying video material of a specific type or content. However, in practice, manual or tag-based screening results in low efficiency in selecting the video material that meets the requirements from a massive amount of material.

[0072] Based on this, the embodiments of the present application provide a video material screening method and device, an electronic device and a storage medium, aiming to improve the efficiency of video material screening.

[0073] The video material screening method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the video material screening method in the embodiments of the present application is described.

[0074] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0075] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] The video material screening method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The video material screening method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video material screening method, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0078] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0079] Figure 1 This is an optional flowchart of the video material screening method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0080] Step S101, obtaining original video material;

[0081] Step S102: extracting video frames from the original video material to obtain an initial video frame sequence;

[0082] Step S103, performing subtitle detection on the initial video frame sequence to obtain subtitle detection data;

[0083] Step S104, performing semantic extraction on the original video material to obtain semantic features of the material;

[0084] Step S105, indexing and constructing based on the subtitle detection data, the material semantic features and the original video material to obtain a video material database;

[0085] Step S106: obtaining a material screening request and performing sentence conversion on the material screening request to obtain a material query sentence;

[0086] Step S107 , screening the video material database based on the material query statement to obtain the target video material.

[0087] In the steps S101 to S107 shown in the embodiment of the present application, the original video material obtained is subjected to video frame extraction to obtain an initial video frame sequence, which facilitates the subsequent subtitle detection and semantic extraction of the video content of the original video material. Next, subtitle detection is performed on the initial video frame sequence to accurately locate the subtitle information and obtain subtitle detection data. At the same time, semantic extraction is performed on the original video material to mine the deep semantic features of the material and obtain the semantic features of the material. Furthermore, an index is constructed based on the subtitle detection data, the semantic features of the material and the original video material to obtain a video material database, so that the video material database has an efficient retrieval capability. Finally, a material screening request is obtained, the material screening request is converted into a sentence to obtain a material query sentence, and the video material database is screened based on the material query sentence. The target video material can be quickly and accurately screened from a large amount of material, saving the time of manual search and improving the efficiency and accuracy of video material screening.

[0088] In step S101 of some embodiments, the original video material is typically a compressed and encoded video file, and the format of the original video material may be H.264, H.265, MPEG-4, etc. The original video material includes picture information, sound, subtitles, etc. The original video material can be collected according to different application scenarios.

[0089] For example, in the FinTech space, you might collect videos of elite groups working in the workplace, aerial footage of cities including office buildings, and financial trend charts. In the Medical Technology space, you might collect images of medical staff, videos of medical equipment, and footage of medical staff conducting consultations. The specific selection depends on the actual application scenario, but is not limited to these.

[0090] In addition, it should be noted that there are multiple original video materials.

[0091] In step S102 of some embodiments, a video decoder is used to extract video frames from the original video material to obtain an initial video frame sequence. The type of video decoder depends on the format of the original video material and is not limited in this embodiment.

[0092] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S205:

[0093] Step S201, performing bitstream parsing on the original video material to obtain video bitstream data;

[0094] Step S202, performing entropy decoding on the video bitstream data to obtain video coding features;

[0095] Step S203, dequantizing the video coding features to obtain video pixel values;

[0096] Step S204, performing inverse transformation processing on the video pixel values ​​to obtain an original video frame sequence;

[0097] Step S205 , grouping the original video frame sequence into frames to obtain an initial video frame sequence.

[0098] In steps S201 to S205 shown in the embodiment of the present application, by performing bitstream parsing on the original video material, the video bitstream data can be accurately extracted. The video bitstream data is entropy decoded to restore the video coding features. The video coding features are dequantized and converted back to video pixel values, which can reduce the impact of quantization errors. Finally, the video pixel values ​​are inversely transformed to obtain the initial video frame sequence of the original video material, and the original video frame sequence is frame grouped to obtain the initial video frame sequence, which ensures the integrity and quality of the video content of the original video material and helps to improve the accuracy of subsequent video frame processing.

[0099] In step S201 of some embodiments, the video decoder first needs to read and parse the input original video material to form video bitstream data. The video bitstream data includes video frame data and some metadata, such as frame type, timestamp, etc.

[0100] In step S202 of some embodiments, entropy decoding is performed on the video bitstream data according to a predefined Huffman coding table to convert the video bitstream data into a state before coding, that is, an original symbol sequence, to obtain video coding features.

[0101] For example, suppose that during the encoding process, a pixel difference value (0) is encoded as a binary codeword of "01". Then during entropy decoding, the decoder will read the two bits of "01" in the bit stream and then convert them back to the original pixel difference value 0 according to the Huffman coding table.

[0102] In addition, arithmetic coding can be used to perform entropy coding to obtain video coding features.

[0103] In step S203 of some embodiments, during the video encoding process, the pixel values ​​of the video frame are mapped to video coding features, i.e., a quantization process. Specifically, the video coding features are a set of discrete values. Therefore, during decoding, the video coding features need to be dequantized to recover the video pixel values.

[0104] In step S204 of some embodiments, during the encoding process, to remove spatial redundancy in the data, the image block is typically transformed, such as a discrete cosine transform, and the inverse transform is to restore the transformed coefficients to the original spatial domain data (pixel values). For the discrete cosine transform, the inverse transform process requires performing an inverse discrete cosine transform calculation on the quantized discrete cosine transform coefficients.

[0105] In step S205 of some embodiments, after decoding to obtain the original video frame sequence, video frames are extracted according to the calculated extraction frame interval to obtain an initial video frame sequence for subsequent subtitle detection. The extraction frame interval is used to limit the extraction of a certain number of frames per second, such as 3 frames or 5 frames.

[0106] Specifically, firstly, the interval of the original video frame sequence is calculated to obtain the extracted frame interval, as shown in formula (1):

[0107]

[0108] Among them, FPS original is the frame rate of the original video frame sequence, FPS target is the target frame rate, and I is the extraction frame interval.

[0109] It should be noted that the target frame rate is a preset value and needs to be set according to the actual application scenario.

[0110] Furthermore, based on the calculated extraction frame interval, the index position of the frame to be extracted in the original video frame sequence is determined by the frame index. If the original video frame sequence starts from index 0, the index of the first frame extracted is 0, and the index of the subsequently extracted frames is 1, 2I, 2I, etc. For example, when I=3, the extracted frame indices are 0, 3, 6, 9, ..., and for the original video frame sequence F0, F1, F2, F3, etc., the extracted initial video frame sequence is F0, F3, F6, etc., that is, the first initial video frame sequence includes F0, F1, F2, the second initial video frame sequence includes F3, F4, F5, and so on. No further details are given.

[0111] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S303:

[0112] Step S301, extracting visual features from the initial video frame sequence to obtain original video visual features;

[0113] Step S302: extracting text features from the initial video frame sequence to obtain original subtitle text features;

[0114] Step S303 , performing subtitle detection based on the original video visual features and the original subtitle text features to obtain subtitle detection data.

[0115] Steps S301 to S303 illustrated in the present embodiment of the present invention perform visual feature extraction on the initial video frame sequence to capture key information such as images, colors, and objects in the video frames, obtaining the original video visual features, which provide a visual foundation for subsequent analysis. Text feature extraction on the initial video frame sequence accurately captures the text content of the subtitles in the video, obtaining the original subtitle text features. Finally, subtitle detection is performed based on the original video visual features and the original subtitle text features to obtain subtitle detection data. This fully utilizes both visual and textual information to more accurately and comprehensively locate subtitles and identify their content, thereby improving the accuracy and reliability of subtitle detection.

[0116] In step S301 of some embodiments, visual features are extracted from the initial video frame sequence using a preset visual feature extraction model to obtain original video visual features.

[0117] Specifically, the preset visual feature extraction model may be a ViT-L / 14 model, a ViT-L / 16 model, a ViT-L / 32 model, etc., but is not limited thereto.

[0118] See also Figure 4 In some embodiments, step S302 may include but is not limited to steps S401 to S404:

[0119] Step S401, performing text region recognition on the initial video frame sequence to obtain text region position information;

[0120] Step S402, performing character recognition on the initial video frame sequence based on the text region position information to obtain original video subtitle data;

[0121] Step S403, performing word segmentation processing on the original video subtitle data to obtain a subtitle word sequence;

[0122] Step S404: feature encoding is performed on the subtitle word sequence to obtain original subtitle text features.

[0123] In steps S401 to S404, as shown in the embodiment of the present application, text region recognition is performed on the initial video frame sequence to determine the location of the subtitles and obtain text region location information, which can provide a precise range for subsequent character recognition. Next, character recognition is performed on the initial video frame sequence based on the text region location information to obtain raw video subtitle data. The identified raw video subtitle data is then segmented to obtain a subtitle word sequence. Finally, feature encoding is performed on the subtitle word sequence to obtain raw subtitle text features, laying a good foundation for subsequent subtitle detection.

[0124] In step S401 of some embodiments, the initial video frame sequence is first preprocessed, and then a preset text detection algorithm (such as EAST, DBNet, PixelLink, etc.) is used to determine the text area bounding box to obtain text area position information.

[0125] In step S402 of some embodiments, character recognition is performed on the initial video frame sequence based on text region position information using a preset character recognition algorithm (OCR model) to obtain original video subtitle data, wherein the original video subtitle data is content in a character string format.

[0126] In step S403 of some embodiments, a preset word segmentation model (such as RoBERTa, BERT, etc.) is used to perform word segmentation processing on the original video subtitle data and encode it into subword units. When mapping it into an embedding vector, position and other embeddings are added together to obtain a subtitle word sequence.

[0127] In step S404 of some embodiments, a preset Transformer encoder is used to perform feature encoding on the subtitle word sequence to obtain original subtitle text features.

[0128] See also Figure 5In some embodiments, step S303 may also include but is not limited to steps S501 to S503:

[0129] Step S501, aligning the original video visual features and the original subtitle text features to obtain an initial alignment score;

[0130] Step S502: performing aggregation calculation on the initial alignment scores based on a preset sliding window to obtain an image-caption alignment score;

[0131] Step S503: performing subtitle recognition based on the image-subtitle alignment score to obtain subtitle detection data.

[0132] In steps S501 to S503, as shown in the embodiment of the present application, an initial alignment score is obtained by aligning the original video visual features with the original subtitle text features. This score can initially clarify the degree of correlation between the visual and text features. Next, the initial alignment score is aggregated based on a preset sliding window to obtain an image-subtitle alignment score. This score can integrate local information, reduce noise interference, and enhance the accuracy of the alignment. Finally, subtitle recognition is performed based on the image-subtitle alignment score to obtain subtitle detection data, which can improve the accuracy and reliability of subtitle recognition.

[0133] In addition, by aligning visual features and text features, we can use inter-modal semantic consistency analysis to achieve more essential subtitle detection, significantly reducing computational complexity while maintaining high accuracy.

[0134] See also Figure 6 In some embodiments, step S501 includes but is not limited to steps S601 to S604:

[0135] Step S601, obtaining a preset difference adjustment parameter;

[0136] Step S602, performing similarity calculation on the original video visual features and the original subtitle text features to obtain feature similarity data;

[0137] Step S603, performing difference adjustment on the feature similarity data based on the difference adjustment parameter to obtain an alignment difference feature;

[0138] Step S604: Activate and calculate the alignment difference features to obtain an initial alignment score.

[0139] Steps S601 to S604 shown in the embodiment of the present application are used to subsequently adjust the differences between visual features and text features by obtaining preset difference adjustment parameters. Furthermore, similarity calculations are performed on the original video visual features and the original subtitle text features to clarify the degree of correlation between the two, obtain feature similarity data, and perform difference adjustment on the feature similarity data based on the difference adjustment parameters to obtain alignment difference features, which can flexibly adapt to the needs of different scenarios and highlight the degree of difference between the two. Finally, the alignment difference features are activated and calculated to obtain an initial alignment score, which can more accurately reflect the matching scores of visual and subtitle self-detection and improve the accuracy of subtitle detection.

[0140] In step S601 of some embodiments, a preset difference adjustment parameter is used to control the sharpness or smoothness of the probability distribution, thereby optimizing the model's performance. Specifically, the difference adjustment parameter needs to be set according to the actual application scenario. It can adjust the sensitivity and accuracy of the alignment based on different video content, subtitle styles, and application scenarios, making the alignment process more targeted and adaptable.

[0141] For example: when the difference between vision and text is small, by setting a larger difference adjustment parameter, the difference between the two can be highlighted, such as setting the difference adjustment parameter to 10; when the difference between vision and text is large, by setting a smaller difference adjustment parameter, the difference between the two can be narrowed, such as setting the difference adjustment parameter to 5.

[0142] In step S602 of some embodiments, similarity calculation can be implemented using methods such as Hamming distance, Euclidean distance, Manhattan distance, and cosine similarity. By performing similarity calculations on the original video visual features and the original subtitle text features, the correlation between the original video visual features and the original subtitle text features can be measured, providing a basis for subsequent alignment operations and helping to accurately determine whether the video and subtitles match.

[0143] In step S603 of some embodiments, the feature similarity data is difference-adjusted based on the difference adjustment parameter. By utilizing the adjustment function of the difference adjustment parameter, attention to key differences in the alignment process can be enhanced, so that the aligned difference features can more accurately reflect the matching between the video and subtitles, thereby improving the accuracy and robustness of the alignment.

[0144] Specifically, this can be achieved by aggregating the difference adjustment parameters and feature similarity data. The calculation process is as follows:

[0145] Alignment difference feature = difference adjustment parameter * feature similarity data.

[0146] In step S604 of some embodiments, a preset activation function is used to activate the alignment difference feature to obtain an initial alignment score. The preset activation function may be a softmax function, a Tanh function, a ReLU function, etc., but is not limited thereto.

[0147] In some embodiments, it is assumed that the initial video frame sequence is denoted as The ViT-L / 14 model is used as the preset visual feature extraction model for the initial video frame sequence Perform visual feature extraction to obtain the original video visual features. The calculation process is as follows:

[0148]

[0149] Among them, d v is the visual embedding dimension, d v =768; V t is the RGB image tensor of the t-th frame, with a resolution of 224×224, is the visual feature of the original video.

[0150] For the initial video frame sequence Perform text feature extraction to obtain the original subtitle text features. The calculation process is as follows:

[0151]

[0152] Among them, T() is the OCR model, T(V t ) is the original video subtitle data, d t is the text embedding dimension, d t = 1024, use a zero vector as a placeholder when the OCR model does not detect text.

[0153] Furthermore, the alignment difference features are calculated based on the preset difference adjustment parameters, the original video visual features, and the original subtitle text features. The calculation process is as follows:

[0154]

[0155] Among them, τ is the preset difference adjustment parameter, is the feature similarity data, is the alignment difference feature, and σ is the activation function.

[0156] In step S502 of some embodiments, since single-frame detection may be unstable, which affects the accuracy of subtitle detection, a preset sliding window is used to perform aggregate calculation on the initial alignment score. The calculation process is as follows:

[0157]

[0158] in, is the image caption alignment score, s i is the initial alignment score of the i-th frame, and N is the size of the sliding window.

[0159] It should be noted that the size of the sliding window needs to be set according to the actual application scenario, for example, the sliding window N is set to 16, 24, etc., but is not limited thereto.

[0160] See also Figure 7 In some embodiments, step S503 may include but is not limited to steps S701 to S703:

[0161] Step S701: performing subtitle classification based on the image-subtitle alignment score to obtain a video frame subtitle type; wherein the video frame subtitle type is used to indicate whether the initial video frame sequence contains subtitles or does not contain subtitles;

[0162] Step S702, performing status identification on the initial video frame sequence based on the video frame subtitle type to obtain a subtitle identification status;

[0163] Step S703 : performing continuous time parsing on the initial video frame sequence based on the subtitle identification status to obtain subtitle detection data.

[0164] In steps S701 to S703, as shown in the embodiment of the present application, subtitles are classified based on the image-subtitle alignment score to identify the video frame subtitle type and distinguish between frame sequences with and without subtitles. Next, the initial video frame sequence is identified based on the video frame subtitle type to obtain a subtitle identification status, which can intuitively mark the status of each frame. Finally, the initial video frame sequence is continuously time-parsed based on the subtitle identification status to obtain subtitle detection data, which can accurately locate the time periods when subtitles appear and disappear, effectively extract subtitle information, and improve the efficiency and accuracy of subtitle detection.

[0165] In some embodiments, step S701 may include but is not limited to the following steps:

[0166] Calculate the image-caption alignment scores and obtain the mean and standard deviation;

[0167] Perform aggregation calculation based on the preset classification factor, mean and standard deviation to obtain the target classification threshold;

[0168] The image caption alignment scores are classified based on the target classification threshold to obtain the video frame caption type.

[0169] Specifically, the preset classification factor is used to adjust the difference between the mean and the standard deviation, and needs to be set according to the distribution of the image-caption alignment score, but is not limited thereto.

[0170] The specific calculation process is as follows:

[0171] γ=μ-αθ (6);

[0172]

[0173] Among them, α is the preset classification factor, μ is the mean, θ is the standard deviation, γ is the target classification threshold, and y t is the video frame subtitle type, where y t =1 means there are subtitles, y t =0 means no subtitles.

[0174] In step S702 of some embodiments, a temporal consistency filter is used to identify the status of the initial video frame sequence based on the video frame subtitle type. The specific process is as follows:

[0175] Assume that the initial state is S0=OUTSIDE, for each y t , if S t-1 =OUTSIDE, and y t =1, enter PENDING state, record the start time t start ; If s t-1 =PENDING,y t =1 and tt start >>k, switch to INSIDE state, confirm there is no subtitle segment y t = 0, fall back to OUTSIDE state. K is the duration of the subtitle continuous state, usually K = 8 to ensure the continuity of the segment.

[0176] It should be noted that the subtitle identification status includes OUTSIDE state, PENDING state and INSIDE state. The OUTSIDE state is used to indicate that no subtitles are currently detected; the PENDING state is used to indicate that subtitles are detected but need to be continuously confirmed; the INSIDE state is used to indicate that subtitles are stably present.

[0177] In step S703 of some embodiments, the initial video frame sequence is continuously analyzed in time based on the subtitle identification state to obtain subtitle detection data, which is used to reflect the time period when the subtitle appears and disappears. Specifically, the intervals of all INSIDE states [t start , t end ], get the time period with subtitles, and the rest of the time period is without subtitles.

[0178] In step S104 of some embodiments, first, each original video material is segmented based on the subtitle detection data to obtain initial video materials, wherein the initial video materials include video materials with subtitles and video materials without subtitles.

[0179] The semantic features of the video material are extracted by using a preset semantic feature extraction model.

[0180] Specifically, the pre-set semantic feature extraction model is a convolutional neural network (CNN). The first layer is a convolutional layer, which uses multiple learnable convolution kernels to slide over the initial video frame sequence, performing convolution operations and extracting low-level features. After the convolution, a ReLU activation function is applied to introduce nonlinearity, enabling the model to learn more complex features. Next, maximum pooling is used to reduce the spatial dimensionality of the feature map. The feature map obtained through multiple layers of convolution, activation, and pooling operations represents higher-level semantic features of the initial video frame sequence, namely, the semantic features of the material.

[0181] It's important to note that while both the semantic feature extraction model and the visual feature extraction model extract visual features from the initial video frame sequence, their model structures are different, and the features they extract are also different. Specifically, the semantic feature extraction model extracts richer, more complete, and more accurate information.

[0182] In step S105 of some embodiments, for each initial video material, an index is constructed based on subtitle detection data and material semantic features to obtain multiple index entries, and the multiple index entries are stored to obtain a video material database.

[0183] The index entries include unique identification information of the original video material (such as video file name + timestamp start and end time), material semantic features, subtitle detection marks and other information.

[0184] It should be noted that the subtitle detection flag is determined based on subtitle detection data. A field, such as a "has_subtitles" field, is added to the index entry of the initial video material to indicate whether there are subtitles.

[0185] If the subtitle detection data indicates that there are subtitles, the "has_subtitles" field is "true", indicating that there are subtitles;

[0186] If the subtitle detection data indicates that there are no subtitles, the "has_subtitles" field is "false", indicating that there are no subtitles.

[0187] It can be understood that, through the specific implementation method illustrated by steps S101 to S105, subtitle detection, semantic extraction, and database construction can be automatically performed on the original video material without manual labeling. This can not only ensure the accuracy of video material processing and save time in processing video materials, but also help improve the efficiency of video material screening.

[0188] In step S106 of some embodiments, when a material screening request generated by a business person or designer is obtained, the material screening request is converted into a material query statement that can be recognized and used by a computer, thereby screening the video material database based on the material query statement to obtain the target video material.

[0189] For example, business personnel and designers enter query text into the search box on the front-end interface to generate a material query request. The back-end system performs preliminary processing on the input query text, including but not limited to removing extra spaces and punctuation marks. The query text is then broken down into individual words using a word segmentation algorithm (maximum matching method (MM)). Part-of-speech tagging is then performed on the segmented words to determine the part of speech of each word in the query text, such as noun, verb, adjective, etc. Semantic analysis technology, combined with context and a semantic knowledge base, is used to understand the overall semantics of the query text. Based on the results of semantic understanding, the query intent of the business personnel and designers is then extracted.

[0190] Next, the extracted intent is converted into conditions that can be searched in the index. Specifically, a material query statement is constructed to search for video materials that meet the conditions in the video material database.

[0191] In this scenario, the generated material query statement may be to search for video materials whose "has_subtitles" field in the index entry is "false".

[0192] In addition, the material query statement can also be used to query the semantic features of the material and generate video materials that filter out matching semantic features.

[0193] In step S107 of some embodiments, the video material database is screened based on the material query statement to obtain a target video material. The number of target video materials is at least one. If multiple target video materials are included, a material list can be generated based on the index entry of each target video material, with each element in the material list corresponding to a target video material. Specifically, each element includes information such as the video material path, video file name, timestamp start and end times, material semantic features, and subtitle detection flags.

[0194] Business personnel and designers can preview the video materials based on the display results in the material list by clicking on the corresponding elements; they can also further filter and sort the material list.

[0195] In addition, the output format can be determined according to the scenario requirements, such as exporting to JSON format or CSV format, but is not limited thereto.

[0196] In some embodiments, the video material screening method provided in the embodiments of the present application can be further designed into a standardized interface, which can be seamlessly integrated into the existing video processing pipeline, and the fully automatic subtitle-free detection function can be achieved by adding only a small amount of computing overhead.

[0197] The video material screening method provided in the embodiment of the present application can be applied to financial technology scenarios or medical technology scenarios. The original video materials corresponding to each scenario are processed and indexed to form a video material database. When the business personnel input a material screening request, the statement conversion is performed based on the material screening request to obtain a material query statement. The video material database is screened based on the material query statement to obtain the target video material that meets the scenario requirements.

[0198] Specifically, it is possible to quickly filter out video materials without subtitles or video materials with matching semantic features from target video materials to meet the needs of producing promotional, marketing, and teaching videos.

[0199] It is understandable that pure video materials, that is, video materials without subtitles, are required for the production of promotional, marketing, and teaching videos. In most cases, the sources of original video materials are diverse, and it is impossible to ensure that each original video material has a high degree of purity. Therefore, the video material screening method provided in the embodiment of the present application can quickly process each original video material, facilitate subsequent material screening, and can quickly and accurately filter out target video materials from massive materials, saving manual search time, improving the efficiency and accuracy of video material screening, and meeting the use and creation needs of video materials in financial technology scenarios and medical technology scenarios.

[0200] See also Figure 8 The embodiment of the present application further provides a video material screening device, which can implement the above-mentioned video material screening method, and the device includes:

[0201] Video material acquisition module 801, used to acquire original video material;

[0202] The video frame extraction module 802 is used to extract video frames from the original video material to obtain an initial video frame sequence;

[0203] The subtitle detection module 803 is used to perform subtitle detection on the initial video frame sequence to obtain subtitle detection data;

[0204] Semantic extraction module 804, used to extract semantics from the original video material to obtain semantic features of the material;

[0205] An index construction module 805 is used to construct an index based on subtitle detection data, material semantic features and original video material to obtain a video material database;

[0206] The query statement conversion module 806 is used to obtain a material screening request and perform statement conversion on the material screening request to obtain a material query statement;

[0207] The target material screening module 807 is used to screen the video material database based on the material query statement to obtain the target video material.

[0208] The specific implementation of the video material screening device is basically the same as the specific embodiment of the above-mentioned video material screening method, and will not be repeated here.

[0209] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned video material screening method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0210] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0211] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0212] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the video material screening method of the embodiments of this application.

[0213] Input / output interface 903, used to implement information input and output;

[0214] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0215] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0216] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0217] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video material screening method is implemented.

[0218] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0219] The video material screening method and device, electronic device and storage medium provided in the embodiments of the present application obtain video frames of the original video material to obtain an initial video frame sequence, which facilitates the subsequent subtitle detection and semantic extraction of the video content of the original video material. Then, subtitle detection is performed on the initial video frame sequence to accurately locate the subtitle information and obtain subtitle detection data. At the same time, semantic extraction is performed on the original video material to mine the deep semantic features of the material and obtain the semantic features of the material. Furthermore, an index is constructed based on the subtitle detection data, the semantic features of the material and the original video material to obtain a video material database, so that the video material database has an efficient retrieval capability. Finally, a material screening request is obtained, the material screening request is converted into a sentence to obtain a material query sentence, and the video material database is screened based on the material query sentence. The target video material can be quickly and accurately screened from a large amount of material, saving the time of manual search and improving the efficiency and accuracy of video material screening.

[0220] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0221] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0222] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0223] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0224] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0225] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0227] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0228] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0229] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0230] The non-Company software tools or components appearing in the embodiments of this application are for illustrative purposes only and do not represent actual use.

[0231] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A video material screening method, characterized in that: The method comprises: Get the original video material; Extracting video frames from the original video material to obtain an initial video frame sequence; Performing subtitle detection on the initial video frame sequence to obtain subtitle detection data; Performing semantic extraction on the original video material to obtain semantic features of the material; An index is constructed based on the subtitle detection data, the material semantic features and the original video material to obtain a video material database; Obtaining a material screening request, and performing sentence conversion on the material screening request to obtain a material query sentence; The video material database is screened based on the material query statement to obtain target video material.

2. The method according to claim 1, characterized in that The performing subtitle detection on the initial video frame sequence to obtain subtitle detection data includes: Extracting visual features from the initial video frame sequence to obtain original video visual features; Performing text feature extraction on the initial video frame sequence to obtain original subtitle text features; Subtitle detection is performed based on the original video visual features and the original subtitle text features to obtain the subtitle detection data.

3. The method according to claim 2, characterized in that The performing subtitle detection based on the original video visual features and the original subtitle text features to obtain the subtitle detection data includes: Performing feature alignment on the original video visual features and the original subtitle text features to obtain an initial alignment score; Performing aggregation calculation on the initial alignment scores based on a preset sliding window to obtain an image-caption alignment score; Subtitle recognition is performed based on the image-subtitle alignment score to obtain the subtitle detection data.

4. The method according to claim 3, characterized in that The step of aligning the original video visual features with the original subtitle text features to obtain an initial alignment score includes: Obtaining preset difference adjustment parameters; Calculating similarity between the original video visual features and the original subtitle text features to obtain feature similarity data; Performing difference adjustment on the feature similarity data based on the difference adjustment parameter to obtain an alignment difference feature; An activation calculation is performed on the alignment difference feature to obtain the initial alignment score.

5. The method according to claim 3, characterized in that The performing subtitle recognition based on the image-subtitle alignment score to obtain the subtitle detection data includes: Performing subtitle classification based on the image subtitle alignment score to obtain a video frame subtitle type; wherein the video frame subtitle type is used to indicate whether the initial video frame sequence contains subtitles or does not contain subtitles; Performing a status identification on the initial video frame sequence based on the video frame subtitle type to obtain a subtitle identification status; The initial video frame sequence is continuously time-parsed based on the subtitle identification state to obtain the subtitle detection data.

6. The method according to claim 2, characterized in that The extracting text features from the initial video frame sequence to obtain original subtitle text features includes: Performing text region recognition on the initial video frame sequence to obtain text region position information; Performing character recognition on the initial video frame sequence based on the text region position information to obtain original video subtitle data; Performing word segmentation processing on the original video subtitle data to obtain a subtitle word sequence; Feature encoding is performed on the subtitle word sequence to obtain the original subtitle text features.

7. The method according to any one of claims 1 to 5, characterized in that The extracting video frames from the original video material to obtain an initial video frame sequence includes: Performing bitstream analysis on the original video material to obtain video bitstream data; Performing entropy decoding on the video bitstream data to obtain video coding features; Dequantizing the video coding features to obtain video pixel values; Performing inverse transformation on the video pixel values ​​to obtain an original video frame sequence; The original video frame sequence is grouped into frames to obtain the initial video frame sequence.

8. A video material screening device, characterized in that: The device comprises: Video material acquisition module, used to obtain original video material; A video frame extraction module, configured to extract video frames from the original video material to obtain an initial video frame sequence; A subtitle detection module, configured to perform subtitle detection on the initial video frame sequence to obtain subtitle detection data; A semantic extraction module, configured to perform semantic extraction on the original video material to obtain semantic features of the material; An index construction module, configured to construct an index based on the subtitle detection data, the material semantic features and the original video material to obtain a video material database; A query statement conversion module is used to obtain a material screening request and perform statement conversion on the material screening request to obtain a material query statement; The target material screening module is used to screen the video material database based on the material query statement to obtain the target video material.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images