A method for understanding unlabeled video content based on AI model construction

Abstract

This invention discloses an AI model-based method for understanding unlabeled video content, belonging to the field of video analysis technology. It addresses the problem of high costs associated with training models on large-scale manually labeled video datasets, hindering multi-dimensional in-depth analysis of long videos. The method includes multi-modal video processing, receiving large local video files uploaded by users, and preprocessing the video files by decoding, frame extraction, and audio separation to generate basic video metadata and the video stream data to be analyzed. This invention combines a multi-modal large language model with professionally optimized prompt word templates to achieve multi-dimensional video analysis, enabling in-depth video understanding in unlabeled scenarios. It eliminates the need for manually labeled datasets, significantly reducing the cost of video analysis. Furthermore, through an unsupervised multi-modal shot boundary detection algorithm, it achieves accurate shot segmentation in unlabeled scenarios, improving the accuracy of shot boundary detection.

Description

Technical Field

[0001] This invention belongs to the field of video analysis technology, specifically relating to a method for understanding unlabeled video content based on AI model construction. Background Technology

[0002] With the rapid development of the digital content industry, the amount of video data has exploded. Mainstream platforms add hundreds of thousands of hours of new content every day. Education, corporate meetings, short videos and other fields have also accumulated massive amounts of data. How to efficiently extract valuable information and achieve automated processing and knowledge organization has become an important direction in the field of artificial intelligence.

[0003] Existing supervised learning-based video analysis techniques heavily rely on large-scale manually annotated video datasets for model training, resulting in high annotation costs and poor model generalization ability. Meanwhile, existing video understanding solutions based on large language models suffer from limitations in input length, interference from redundant information, and insufficient fine-grained semantic capture, making it difficult to achieve multi-dimensional in-depth analysis of long videos. Furthermore, existing video analysis tools are functionally limited, mostly only capable of basic functions such as shot segmentation, face recognition, and speech-to-text conversion, failing to provide the professional-grade functions required by the film and television industry, such as script reverse engineering, professional analysis of cinematic feel, in-depth detection of cinematic language, and systematic analysis of emotional tone. These features fail to meet the core needs of film and television professionals, content creators, and professional video analysts.

[0004] Therefore, there is a need for an unlabeled video content understanding method based on AI model construction to address the problems of existing technologies that rely heavily on large-scale manually labeled video datasets for model training, resulting in high labeling costs, difficulty in achieving multi-dimensional in-depth analysis of long videos, and limited functionality of video analysis tools, which cannot provide the professional-grade functions required by the film and television industry. Summary of the Invention

[0005] The purpose of this invention is to provide a method for understanding unlabeled video content based on AI model construction, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for understanding unlabeled video content based on an AI model, comprising:

[0007] Step S1: Multi-mode video processing. Receive large local video files uploaded by users, perform preprocessing on the video files including decoding, frame extraction, and audio separation, and generate basic video metadata and video stream data to be analyzed.

[0008] Step S2: Based on professional-grade AI engine processing and analysis, call the multimodal large language model, and based on the built-in multiple sets of professional prompt word templates, perform unlabeled multi-dimensional in-depth analysis on the preprocessed video stream data to generate a structured analysis report containing timestamp anchors.

[0009] Step S3: Build an immersive interactive experience, based on the structured analysis report, to achieve linked jumps between timestamps and video players, contextual interactive Q&A based on the full video content and analysis results, and real-time streaming rendering output of analysis results;

[0010] Step S4: Adaptive architecture adaptation. Based on the type of access terminal, automatically adapt the dual interactive interface of the mobile terminal bottom operation area optimization layout and the desktop terminal sidebar management layout.

[0011] Step S5, Security and Authentication Management: Instance access permissions are managed through an IP whitelist access control mechanism, while high availability of the analysis service is ensured through automatic switching of multi-service provider multi-modal model APIs and load balancing mechanisms.

[0012] It should be noted in the solution that, in step S2, the built-in multiple sets of professional prompt word templates include script reverse engineering templates, cinematic feel analysis templates, shot language detection templates, and emotional tone analysis templates; the script reverse engineering template has built-in Hollywood standard script format specifications, which is used to drive the multimodal large language model to restore the video content into a script that conforms to Hollywood standards, and the output content includes scene numbers, interior and exterior scene identifiers, time markers, location information, character dialogue, action descriptions, shot annotations, and corresponding start and end time stamps; the cinematic feel analysis template has built-in professional film and television production analysis dimensions, which is used to drive the model to analyze the video. The system performs frame-by-frame analysis, outputting professional analysis content and corresponding timestamps for the composition design, lighting arrangement, color matching, and editing logic of each keyframe. The shot language detection template has a built-in film and television shot language classification system, which drives the model to complete video shot segmentation, identify the shot size, shooting angle, and camera movement of each shot, and output the start and end timestamps and attribute labels of the corresponding shot. The emotional tone analysis template drives the model to combine video content and audio information to analyze the emotional atmosphere and narrative tone created by the audio-visual combination, and outputs analysis content on the emotional type, intensity changes, and atmosphere creation logic of the corresponding time interval.

[0013] It is further worth noting that the lens language detection template identifies the framing, shooting angle, and camera movement of each shot. An unsupervised multimodal lens boundary detection algorithm is used to achieve accurate lens segmentation in unlabeled scenes. The specific implementation steps and formulas of the algorithm are as follows:

[0014] S2.1 Extract a sequence of consecutive video frames from the preprocessed video stream at a preset frame rate. Simultaneously, extract the audio feature sequence of the corresponding frame interval. , where n is the total number of video frames, and the duration of each audio segment corresponding to each frame is 1 / preset frame rate in seconds;

[0015] S2.2, For each frame of image Frame-level visual features are extracted using a pre-trained visual Transformer model. Where d is the feature dimension, with a default value of 768; for audio segments Frame-level audio features are extracted using a pre-trained audio Transformer model. To ensure that the visual and audio features have the same dimensionality, L2 normalization is performed on both visual and audio features to obtain normalized features.

[0016] , ;

[0017] in, It is an L2 norm;

[0018] S2.3 Calculate the multimodal feature difference between adjacent frames, construct the inter-frame difference matrix, and calculate the inter-frame difference. The calculation formula is:

[0019]

[0020] in, This is the visual feature weight coefficient, with a value range of [0,1] and a default value of 0.7; The cosine similarity of visual features between adjacent frames. Cosine similarity of audio features between adjacent frames;

[0021] S2.4 Calculate the adaptive difference threshold T. The calculation formula is as follows:

[0022]

[0023] in, This represents the mean of the total inter-frame differences. is the standard deviation of the full inter-frame difference, and k is the sensitivity coefficient, with a value range of [2,5] and a default value of 3;

[0024] S2.5, Traverse all adjacent frames, when When the location is determined to be the edge of the shot, the video shot is segmented. For each segmented shot, the key frame sequence and corresponding audio content are extracted and input into a multimodal large language model. Based on the shot language detection template, the shot size, shooting angle and camera movement are identified and labeled, and a structured shot labeling report is generated.

[0025] Furthermore, it should be noted that in step S3, the linkage between the timestamp and the video player is implemented as follows: when rendering the structured analysis report, the time intervals corresponding to all analysis content are encoded into interactive timestamp tags with built-in jump parameters. The user's click event is listened to through JavaScript. When triggered, the jump parameters are extracted and a seek command is sent to the video player to control the player to automatically jump to the corresponding time point and play.

[0026] As a preferred implementation, in step S3, the contextual interactive question-and-answer is implemented as follows: the full video metadata, structured analysis report, and full audio-to-text text are stored in a vector database and a text embedding vector is generated; when a user asks a question, the question content is first vectorized, and the top 10 relevant contextual contents are matched by vector similarity retrieval. The retrieved content, historical dialogue records, current question, and question-and-answer prompt word template are concatenated and input into a multimodal large language model to generate a question-and-answer reply. At the same time, the timestamp tag of the corresponding content is automatically matched in the reply.

[0027] As a preferred implementation, in step S3, the real-time streaming rendering output is implemented as follows: using the SSE server to send event protocol, after the client initiates an analysis request, a long connection is established with the server. For each token of analysis content generated by the multimodal large language model, the server immediately pushes it to the client through the long connection, and the client renders the received content in real time.

[0028] In a preferred implementation, the IP whitelist access control in step S5 is implemented as follows: when the server starts, it reads the list of allowed IP addresses configured in the environment variables, supports single IP and CIDR network segment formats, and constructs IP access rules; when a client access request is received, the client's source IP address is extracted and matched with the whitelist list. The request is allowed only if the IP address is within the whitelist range; otherwise, access is denied.

[0029] Compared with existing technologies, the unlabeled video content understanding method based on AI model construction provided by this invention has at least the following beneficial effects:

[0030] By combining a multimodal large language model with professionally optimized prompt word templates, multi-dimensional video analysis can be completed, achieving deep understanding of videos in unlabeled scenarios. This eliminates the need for manually labeled datasets, significantly reducing the cost of video analysis. At the same time, through an unsupervised multimodal shot boundary detection algorithm, accurate shot segmentation in unlabeled scenarios is achieved, improving the accuracy of shot boundary detection and significantly enhancing the model's generalization ability. It can be adapted to various video types and custom analysis scenarios.

[0031] With four built-in professional prompt word templates, it can simultaneously realize Hollywood standard script restoration, frame-by-frame cinematic analysis, full-dimensional detection of camera language, and emotional tone analysis of audio-visual integration, meeting the professional analysis needs of film and television practitioners and content creators in one stop, and greatly improving the depth and professionalism of video analysis.

[0032] The analysis report and video playback are seamlessly integrated through timestamp linkage, allowing users to locate the corresponding video segment with a click; the interactive question-and-answer based on the full video content is realized through a contextual dialogue engine, improving the accuracy of question and answer responses; and the analysis results are rendered in real time through SSE streaming output, eliminating the need for users to wait for the full results to be generated, greatly improving efficiency and user experience. Attached Figure Description

[0033] Figure 1 This is a flowchart of the unlabeled video content understanding method based on AI model construction according to the present invention;

[0034] Figure 2 This is a flowchart of the unsupervised multimodal lens boundary detection algorithm of the present invention;

[0035] Figure 3 This is a structural diagram of the electronic device proposed in this invention;

[0036] Figure 4 This is a schematic diagram of the structure of the computer-readable storage medium proposed in this invention. Detailed Implementation

[0037] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0038] Reference Figure 1 As shown, this invention provides a method for understanding unlabeled video content based on an AI model, including:

[0039] S1. Multi-mode video processing: Receives large local video files uploaded by users, performs preprocessing on the video files including decoding, frame extraction, and audio separation, and generates basic video metadata and video stream data to be analyzed.

[0040] The multi-mode video processing also includes an advanced configuration option processing flow: receiving the start / end time, frame rate adjustment parameters, and additional custom commands of the user-input custom analysis segment, and performing preprocessing such as cropping and re-extracting frames on the video stream data based on the parameters to generate the video stream data to be analyzed for the corresponding custom interval; for local video files larger than 100MB, a segmented upload and streaming preprocessing mechanism is adopted to split the video file into several segments of fixed size of 10MB, and the client uploads the segments in parallel. The server immediately performs decoding, frame extraction, and metadata extraction operations on the received segments without waiting for the entire video file to be uploaded. After all segments are processed, they are spliced ​​together to generate the complete video stream data to be analyzed, supporting video processing of a maximum single file size of 20GB;

[0041] Understandably, in the segmented upload and streaming preprocessing mechanism, the client generates a unique segment sequence number and MD5 checksum for each segment. After receiving the segment, the server verifies the integrity of the segment using the checksum to prevent segment transmission corruption. After all segments are uploaded, the server concatenates all preprocessing results according to the segment sequence number to generate complete video basic metadata, including video resolution, original frame rate, total duration, encoding format, and audio sampling rate information.

[0042] Through segmented upload and streaming preprocessing mechanisms, it supports local large video file processing up to 20GB, solving the problems of stuttering and upload failure in large file processing of existing technologies; at the same time, it supports custom analysis segments, frame rate adjustment and personalized commands, which can meet users' customized analysis needs and adapt to different balance requirements of precision and efficiency.

[0043] S2. Based on professional-grade AI engine processing and analysis, it calls multimodal large language model and uses multiple sets of built-in professional prompt word templates to perform unlabeled multi-dimensional in-depth analysis on preprocessed video stream data, generating a structured analysis report containing timestamp anchors.

[0044] Among them, the built-in professional prompt word templates include script reverse engineering template, cinematic feel analysis template, shot language detection template, and emotional tone analysis template;

[0045] The script reverse engineering template has a built-in Hollywood standard script format specification, which is used to drive a multimodal large language model to restore video content into a script that conforms to Hollywood standards. The output content includes scene number, interior and exterior scene identifiers, time stamps, location information, character dialogue, action descriptions, shot annotations and corresponding start and end timestamps.

[0046] The cinematic analysis template has built-in professional film and television production analysis dimensions to drive the model to break down the video frame by frame and output professional analysis content and corresponding timestamps of the composition design, lighting arrangement, color matching, and editing logic of the corresponding key frames.

[0047] The shot language detection template has a built-in film and television shot language classification system, which is used to drive the model to complete video shot segmentation, identify the shot size, shooting angle and camera movement of each shot, and output the start and end timestamps and attribute labels of the corresponding shot.

[0048] Among them, reference Figure 2 As shown, the framing, shooting angle, and camera movement of each shot are identified. An unsupervised multimodal shot boundary detection algorithm is used to achieve accurate shot segmentation in unlabeled scenes. The specific implementation steps and formulas of the algorithm are as follows:

[0049] S2.1 Extract a sequence of consecutive video frames from the preprocessed video stream at a preset frame rate. Simultaneously, extract the audio feature sequence of the corresponding frame interval. , where n is the total number of video frames, and the duration of each audio segment corresponding to each frame is 1 / preset frame rate in seconds;

[0050] S2.2, For each frame of image Frame-level visual features are extracted using a pre-trained visual Transformer model. Where R is the set of real numbers and d is the feature dimension, with a default value of 768; for audio segments Frame-level audio features are extracted using a pre-trained audio Transformer model. To ensure that the visual and audio features have the same dimensionality, L2 normalization is performed on both visual and audio features to obtain normalized features.

[0051] , ;

[0052] in, It is an L2 norm;

[0053] S2.3 Calculate the multimodal feature difference between adjacent frames, construct the inter-frame difference matrix, and calculate the inter-frame difference. The calculation formula is:

[0054]

[0055] in, This is the visual feature weight coefficient, with a value range of [0,1] and a default value of 0.7; The cosine similarity of visual features between adjacent frames. Cosine similarity of audio features between adjacent frames;

[0056] S2.4 Calculate the adaptive difference threshold T. The calculation formula is as follows:

[0057]

[0058] in, This represents the mean of the total inter-frame differences. is the standard deviation of the full inter-frame difference, and k is the sensitivity coefficient, with a value range of [2,5] and a default value of 3;

[0059] S2.5, Traverse all adjacent frames, when When the location is determined to be the edge of the shot, the video shot is segmented. For each segmented shot, the key frame sequence and corresponding audio content are extracted and input into a multimodal large language model. Based on the shot language detection template, the shot size, shooting angle and camera movement are identified and labeled, and a structured shot labeling report is generated.

[0060] The emotional tone analysis template is used to drive the model to combine video content and audio information to analyze the emotional atmosphere and narrative tone created by the combination of sound and image, and output the analysis content of emotional type, intensity change and atmosphere creation logic in the corresponding time interval;

[0061] S3. Build an immersive interactive experience, based on structured analysis reports, to achieve linked jumps between timestamps and video players, contextual interactive Q&A based on full video content and analysis results, and real-time streaming rendering output of analysis results;

[0062] The implementation of the linkage between timestamps and video player is as follows: When rendering the structured analysis report, the time intervals corresponding to all analysis content are encoded into interactive timestamp tags with built-in jump parameters. JavaScript listens for user click events, and when triggered, the jump parameters are extracted and a seek command is sent to the video player to control the player to automatically jump to the corresponding time point and play the video.

[0063] The implementation of context-based interactive question answering is as follows: all video metadata, structured analysis reports, and audio-to-text full text are stored in a vector database and text embedding vectors are generated; when a user asks a question, the question content is first vectorized, and the top 10 relevant contextual contents are matched by vector similarity retrieval. The retrieved content, historical dialogue records, current question and question-answer prompt word template are concatenated and input into a multimodal large language model to generate a question-answer reply. At the same time, the timestamp tag of the corresponding content is automatically matched in the reply.

[0064] The real-time streaming rendering output is implemented as follows: the SSE server sends an event protocol. After the client initiates an analysis request, a long connection is established with the server. For each token generated by the multimodal large language model, the server immediately pushes the analysis content to the client through the long connection. The client renders the received content in real time without waiting for the full analysis results to be generated.

[0065] S4. Adaptive architecture adaptation: Based on the type of access terminal, it automatically adapts to the dual interactive interface of the mobile terminal bottom operation area optimization layout and the desktop terminal sidebar management layout.

[0066] The specific implementation of the adaptive architecture adaptation steps is as follows: The terminal recognition engine identifies the device type, operating system, and screen parameters of the accessing terminal based on the User-Agent information in the HTTP request header, the screen size collected by the front end, and the touch event support. When a mobile device is identified, a mobile-specific UI layout is automatically loaded, fixing the core operation controls, analysis report switching controls, and dialog input boxes in the bottom operation area of ​​the page, adapting to the range of single-handed thumb operation. The main area adopts a top-bottom layout, with a video player at the top and an analysis report / dialogue window at the bottom. When a desktop device is identified, a desktop-specific UI layout is automatically loaded, setting a sidebar management module on the left side of the page, integrating video file management, analysis template selection, advanced configuration, and historical analysis record functions. The main area adopts a left-right split layout, with a video player on the left and an analysis report / dialogue window on the right, supporting drag-and-drop adjustment of the column width.

[0067] Understandably, the adaptive architecture can automatically identify the terminal type and adapt to the bottom layout optimized for one-handed operation on mobile devices and the sidebar layout optimized for efficient operation on large desktop screens, thus achieving a consistent and optimized user experience across devices.

[0068] S5, security and authentication management, manages instance access permissions through an IP whitelist access control mechanism, and ensures high availability of analysis services through automatic switching of multi-service provider multi-modal model APIs and load balancing mechanisms;

[0069] The IP whitelist access control is implemented as follows: when the server starts, it reads the list of allowed IP addresses configured in the environment variables, supports single IP and CIDR network segment formats, and constructs IP access rules; when a client access request is received, the client's source IP address is extracted and matched with the whitelist list. The request is allowed only if the IP address is within the whitelist range, otherwise access is denied.

[0070] The implementation of multi-service provider backup and load balancing is as follows: pre-configure multimodal model API interfaces from multiple cloud AI service providers and locally privately deployed multimodal model API interfaces to build an API resource pool, and set priority, weight, and health check parameters for each interface; during service operation, perform a health check on all API interfaces every 10 seconds, and update the interface availability status, response latency, and remaining quota in real time; when a user initiates an analysis request, the request is distributed based on priority and weight. When the primary interface fails to call, times out, or exhausts its quota, it is automatically marked as unavailable and switched to the backup interface. At the same time, the request distribution weight is dynamically adjusted based on the response latency to achieve load balancing.

[0071] Understandably, the IP whitelist access control mechanism effectively prevents unauthorized access to privately deployed instances, ensuring data and service security. The automatic switching and load balancing mechanism between multiple service provider APIs solves the problem of dependency on a single API interface, improves service availability, and significantly enhances service stability and resilience.

[0072] In summary, the advantages of this invention are:

[0073] By combining a multimodal large language model with professionally optimized prompt word templates, multi-dimensional video analysis can be completed, achieving deep understanding of videos in unlabeled scenarios. This eliminates the need for manually labeled datasets, significantly reducing the cost of video analysis. At the same time, through an unsupervised multimodal shot boundary detection algorithm, accurate shot segmentation in unlabeled scenarios is achieved, improving the accuracy of shot boundary detection and significantly enhancing the model's generalization ability. It can be adapted to various video types and custom analysis scenarios.

[0074] With four built-in professional prompt word templates, it can simultaneously realize Hollywood standard script restoration, frame-by-frame cinematic analysis, full-dimensional detection of camera language, and emotional tone analysis of audio-visual integration, meeting the professional analysis needs of film and television practitioners and content creators in one stop, and greatly improving the depth and professionalism of video analysis.

[0075] The analysis report and video playback are seamlessly integrated through timestamp linkage, allowing users to locate the corresponding video segment with a click; the interactive question-and-answer based on the full video content is realized through a contextual dialogue engine, improving the accuracy of question and answer responses; and the analysis results are rendered in real time through SSE streaming output, eliminating the need for users to wait for the full results to be generated, greatly improving efficiency and user experience.

[0076] Furthermore, the method according to the embodiments of this application can also be achieved by means of... Figure 3 The architecture of the electronic device shown is used to implement this. For example... Figure 3 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store the unlabeled video content understanding method based on AI model construction provided in this application. The electronic device 500 may also include a user interface 508. Of course, Figure 3 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 3 One or more components in the illustrated electronic device.

[0077] Figure 4 This is a schematic diagram of a computer-readable storage medium structure provided in one embodiment of this application. Figure 4 The diagram illustrates a computer-readable storage medium 600 according to one embodiment of this application. The computer-readable storage medium 600 stores computer-readable instructions. When executed by a processor, the computer-readable instructions can perform an unlabeled video content understanding method based on an AI model construction, as described above with reference to the accompanying drawings, according to an embodiment of this application. The storage medium 600 includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0078] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. An unlabeled video content understanding method based on an AI model construction, characterized in that, include: Step S1: Multi-mode video processing. Receive large local video files uploaded by users, perform preprocessing on the video files including decoding, frame extraction, and audio separation, and generate basic video metadata and video stream data to be analyzed. Step S2: Based on professional-grade AI engine processing and analysis, call the multimodal large language model, and based on the built-in multiple sets of professional prompt word templates, perform unlabeled multi-dimensional in-depth analysis on the preprocessed video stream data to generate a structured analysis report containing timestamp anchors. Step S3: Build an immersive interactive experience, based on the structured analysis report, to achieve linked jumps between timestamps and video players, contextual interactive Q&A based on the full video content and analysis results, and real-time streaming rendering output of analysis results; Step S4: Adaptive architecture adaptation. Based on the type of access terminal, automatically adapt the dual interactive interface of the mobile terminal bottom operation area optimization layout and the desktop terminal sidebar management layout. Step S5, Security and Authentication Management: Instance access permissions are managed through an IP whitelist access control mechanism, while high availability of the analysis service is ensured through automatic switching of multi-service provider multi-modal model APIs and load balancing mechanisms.

2. The method for understanding unlabeled video content based on an AI model as described in claim 1, characterized in that: In step S2, the built-in multiple sets of professional prompt word templates include script reverse engineering templates, cinematic feel analysis templates, shot language detection templates, and emotional tone analysis templates; The script reverse engineering template has a built-in Hollywood standard script format specification, which is used to drive a multimodal large language model to restore the video content into a script that conforms to Hollywood standards. The output content includes scene number, interior and exterior scene identifiers, time identifiers, location information, character dialogue, action descriptions, shot labels and corresponding start and end timestamps. The cinematic feel analysis template incorporates professional film and television production analysis dimensions to drive the model to break down the video frame by frame, outputting professional analysis content and corresponding timestamps for the composition design, lighting arrangement, color matching, and editing logic of the corresponding keyframes; the shot language detection template incorporates a film and television shot language classification system to drive the model to complete video shot segmentation, identify the shot size, shooting angle, and camera movement of each shot, and output the start and end timestamps and attribute labels of the corresponding shot. The emotional tone analysis template is used to drive the model to combine video content and audio information to analyze the emotional atmosphere and narrative tone created by the combination of sound and image, and output the analysis content of emotional type, intensity change and atmosphere creation logic in the corresponding time interval.

3. The method for understanding unlabeled video content based on AI model construction according to claim 2, characterized in that: The lens language detection template identifies the framing, shooting angle, and camera movement of each shot. An unsupervised multimodal lens boundary detection algorithm is used to achieve accurate lens segmentation in unlabeled scenes. The specific implementation steps and formulas of the algorithm are as follows: S2.1 Extract a sequence of consecutive video frames from the preprocessed video stream at a preset frame rate. Simultaneously, extract the audio feature sequence of the corresponding frame interval. , where n is the total number of video frames, and the duration of each audio segment corresponding to each frame is 1 / preset frame rate in seconds; S2.2, For each frame of image Frame-level visual features are extracted using a pre-trained visual Transformer model. Where d is the feature dimension; for audio segments Frame-level audio features are extracted using a pre-trained audio Transformer model. To ensure that the visual and audio features have the same dimensionality, L2 normalization is performed on both visual and audio features to obtain normalized features. ， ； in, It is an L2 norm; S2.3 Calculate the multimodal feature difference between adjacent frames, construct the inter-frame difference matrix, and calculate the inter-frame difference. The calculation formula is: in, This is the visual feature weight coefficient, with a value range of [0,1] and a default value of 0.7; The cosine similarity of visual features between adjacent frames. Cosine similarity of audio features between adjacent frames; S2.4 Calculate the adaptive difference threshold T. The calculation formula is as follows: in, This represents the mean of the total inter-frame differences. is the standard deviation of the full inter-frame difference, and k is the sensitivity coefficient, with a value range of [2,5] and a default value of 3; S2.5, Traverse all adjacent frames, when When the location is determined to be the edge of the shot, the video shot is segmented. For each segmented shot, the key frame sequence and corresponding audio content are extracted and input into a multimodal large language model. Based on the shot language detection template, the shot size, shooting angle and camera movement are identified and labeled, and a structured shot labeling report is generated.

4. The method for understanding unlabeled video content based on AI model construction according to claim 3, characterized in that: In step S3, the linkage between the timestamp and the video player is implemented as follows: when rendering the structured analysis report, all time intervals corresponding to the analysis content are encoded into interactive timestamp tags with built-in jump parameters. The user's click event is listened to through JavaScript. When triggered, the jump parameters are extracted and a seek command is sent to the video player to control the player to automatically jump to the corresponding time point and play.

5. The method for understanding unlabeled video content based on an AI model as described in claim 4, characterized in that: In step S3, the context-based interactive question answering is implemented by storing the full video metadata, structured analysis report, and full audio-to-text text into a vector database and generating text embedding vectors. When a user asks a question, the question content is first vectorized. The top 10 relevant contextual content are then matched using vector similarity. The retrieved content, historical dialogue records, current question and question-answer prompt word templates are concatenated and then input into a multimodal large language model to generate a question-answer reply. At the same time, the timestamp tag of the corresponding content is automatically matched in the reply.

6. The method for understanding unlabeled video content based on AI model construction according to claim 5, characterized in that: In step S3, the real-time streaming rendering output is implemented as follows: using the SSE server to send event protocol, after the client initiates an analysis request, a long connection is established with the server. For each token of analysis content generated by the multimodal large language model, the server immediately pushes it to the client through the long connection, and the client renders the received content in real time.

7. The method for understanding unlabeled video content based on AI model construction according to claim 3, characterized in that: In step S5, the IP whitelist access control is implemented as follows: when the server starts, it reads the list of allowed IP addresses configured in the environment variables, supporting single IP and CIDR network segment formats; Build IP access rules. When a client access request is received, first extract the client's source IP address and match it with the whitelist. Only allow the request if the IP address is within the whitelist range; otherwise, deny access.

8. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform an unlabeled video content understanding method based on an AI model as described in any one of claims 1-7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the unlabeled video content understanding method based on AI model construction as described in any one of claims 1-7.

Images