Data processing method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a data processing method and device based on artificial intelligence, computer equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of video data, and obtaining a feature vector; performing similarity calculation on the feature vectors, and performing dynamic planning segmentation on the video data based on a similarity result so as to divide the video data into video clips; performing time sequence merging processing on the video clips to obtain a video sequence; compressing the video sequence to obtain a visual token sequence; performing importance evaluation on the visual token sequence by using a large language model to obtain an importance evaluation result; performing fusion operation on the visual token sequence based on an importance evaluation result to obtain processing data; and performing reasoning processing on the processing data and the question data based on a large language model to generate answer data, and outputting the answer data. The method can be applied to reply processing scenes in the financial science and technology field and the digital medical field, the calculation cost in the video data processing process can be effectively reduced, and the reasoning speed is increased.

Description

Data processing methods, devices, computer equipment, and media based on artificial intelligence Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to the fields of fintech and digital healthcare, particularly to data processing methods, devices, computer equipment, and storage media based on artificial intelligence. Background Technology

[0002] With the continuous evolution of multimodal large model technology, Video Large Language Model (VideoLLM) has emerged as a promising candidate for understanding complex video content, demonstrating remarkable potential. It can deeply analyze various types of information within videos, bringing innovative applications to numerous industries. However, video data processing faces severe computational bottlenecks. Unlike text data, a video lasting only a few minutes often contains tens of thousands of frames, and each frame, after being processed by the encoder, is transformed into hundreds of visual tokens. As vector representations of video frames, visual tokens can number in the tens of thousands for a single video. This massive number of visual tokens places a huge burden on the reasoning process of large models, directly leading to a significant increase in computational costs and a marked decrease in inference speed, severely restricting the widespread application and efficient operation of Video Large Language Model in real-world scenarios.

[0003] In the financial and insurance sectors, the application of video data is becoming increasingly widespread. For example, in insurance claims investigations, a large amount of accident scene video needs to be processed. These videos record various details of the accident, but contain a lot of redundant information, such as repeated scene shots and the movement of irrelevant personnel. Traditional processing methods require analyzing every video frame one by one, which not only consumes a lot of computing resources but also results in slow reasoning speed, making it difficult to quickly and accurately determine accident liability and loss. For example, in the processing of vehicle collision accident claims videos, there may be multiple segments of repeatedly filmed still vehicle footage. Traditional methods cannot efficiently remove this redundant information, leading to a prolonged claims process, impacting customer experience and the operational efficiency of insurance companies. Furthermore, in the medical field, video data is also of great value, such as surgical procedure videos. Doctors need to extract key information from these videos to assess surgical outcomes and summarize lessons learned. However, surgical videos are usually long and contain many similar or repetitive surgical procedures. Traditional processing methods struggle to quickly locate key surgical steps and important details, increasing the difficulty and time cost for doctors to analyze the videos. For example, in a complex surgical video, there may be multiple surgical operations from similar angles. Traditional methods cannot intelligently filter out representative segments, requiring doctors to spend a lot of time browsing repetitive content when reviewing the video. This is not conducive to the rapid transmission of medical knowledge and the improvement of surgical quality.

[0004] Therefore, how to effectively reduce the computational cost in the video data processing process and improve the inference speed has become a key issue that urgently needs to be addressed in the development of large video language models. Summary of the Invention

[0005] The purpose of this application is to propose a data processing method, apparatus, computer device, and storage medium based on artificial intelligence, so as to solve the technical problems of computational cost and slow inference speed in the existing video large language model for processing video data.

[0006] Firstly, an artificial intelligence-based data processing method is provided, comprising: loading a target page with pre-generated tracking points based on resource location tracking rules; calling a target interface corresponding to the target page; collecting business tracking point data corresponding to target resource locations in the target page from users within a preset time period based on the target interface; generating product conversion data corresponding to the target resource locations based on the business tracking data; and performing recommendation and replacement processing on products within the target resource locations based on the product conversion data.

[0007] Secondly, an artificial intelligence-based data processing device is provided, comprising: a first processing module for receiving input video data and question data, and extracting features from the video data to obtain corresponding feature vectors; a second processing module for calculating similarity between the feature vectors to obtain similarity results, and performing dynamic programming segmentation on the video data based on the similarity results to divide the video data into corresponding video segments; a merging module for performing temporal merging on the video segments to obtain corresponding video sequences; a compression module for compressing the video sequences based on a preset token compression strategy to obtain corresponding visual token sequences; an evaluation module for evaluating the importance of the visual token sequences using a preset large language model based on an attention mechanism to obtain corresponding importance evaluation results; a fusion module for performing fusion operations on the visual token sequences using a preset similarity fusion strategy based on the importance evaluation results to obtain corresponding processed data; and a third processing module for performing reasoning processing on the processed data and the question data based on the large language model to generate corresponding answer data, and performing output processing on the answer data.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described artificial intelligence-based data processing method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned artificial intelligence-based data processing method.

[0010] In the aforementioned scheme implemented by the artificial intelligence-based data processing method, apparatus, computer equipment, and storage medium, the input video data and question data are first received, and feature extraction is performed on the video data to obtain corresponding feature vectors. Then, similarity calculation is performed on the feature vectors to obtain similarity results, and dynamic programming segmentation is performed on the video data based on the similarity results to divide the video data into corresponding video segments. Next, the video segments are temporally merged to obtain corresponding video sequences. Subsequently, the video sequences are compressed based on a preset token compression strategy to obtain corresponding visual token sequences. Based on an attention mechanism, a preset large language model is used to evaluate the importance of the visual token sequences to obtain corresponding importance evaluation results. Further, based on the importance evaluation results, a preset similarity fusion strategy is used to fuse the visual token sequences to obtain corresponding processed data. Finally, the processed data and question data are reasoned together using the large language model to generate corresponding answer data, and the answer data is output. Based on the above automated processing flow, this application adopts a collaborative processing strategy of global planning followed by local refinement. By combining global spatiotemporal compression outside the model with refined information fusion inside the model, it achieves fast and high-precision understanding and question-answering of video data under limited computing resources. This greatly improves the efficiency of video understanding, effectively reduces the computational cost in the video data processing process, increases inference speed, and ensures the accuracy of the generated answer data. Attached Figure Description

[0011] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 is an exemplary system architecture diagram in which this application can be applied; Figure 2 is a flowchart of an embodiment of the artificial intelligence-based data processing method according to this application; Figure 3 is a structural schematic diagram of an embodiment of the artificial intelligence-based data processing apparatus according to this application; Figure 4 is a structural schematic diagram of an embodiment of the computer device according to this application. Detailed Implementation

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] As shown in Figure 1, the system architecture 100 may include a terminal device 101, a network 102, and a server 103. The terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. The network 102 is used as a medium to provide a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0017] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0018] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0019] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0020] It should be noted that the artificial intelligence-based data processing method provided in the embodiments of this application is generally executed by a server / terminal device, and correspondingly, the artificial intelligence-based data processing device is generally set in the server / terminal device.

[0021] It should be understood that the number of terminal devices, networks, and servers shown in Figure 1 is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0022] Referring again to Figure 2, a flowchart of an embodiment of the AI-based data processing method according to this application is shown. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different needs. The AI-based data processing method provided in this application embodiment can be applied to any scenario requiring response processing, and therefore can be applied to products in these scenarios, such as response processing products in the financial insurance field. The AI-based data processing method includes the following steps: Step S201, receiving input video data and question data, and performing feature extraction on the video data to obtain corresponding feature vectors.

[0023] In this embodiment, the electronic device (e.g., the server / terminal device shown in Figure 1) on which the artificial intelligence-based data processing method runs can acquire the input video data and question data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods. The executing entity of this application is specifically a data processing system, also known as a question-answering system, which can be simply referred to as the system. Feature extraction can be performed on each frame of the original video data using a visual encoder. A visual encoder is a trained neural network model that can convert an image into a set of fixed-dimensional feature vectors, which can effectively represent the content and semantic information of the image. For example, for an image containing people and a scene, a visual encoder can extract the features of the people, the features of the scene, and the relationships between them. Furthermore, the aforementioned question data can be questions input by the user based on actual needs, posed to be answered regarding the video data.

[0024] This application can be applied to response processing scenarios in the fintech and digital healthcare fields. For example, in the insurance claims investigation video processing scenario within the financial insurance sector, the input video data may include: a video recording a vehicle accident scene, containing footage of the moment of collision, the vehicle's stationary state after the collision, the surrounding environment (such as road conditions and traffic signs), and scenes of unrelated personnel moving around. The stationary state of the vehicle may consist of multiple repeated shots, such as views of the stationary vehicle's exterior taken from different angles, exhibiting a certain continuity and repetition in time. The input question data may include: "Please determine the main damaged parts of the vehicle and the approximate direction of travel of the vehicle at the time of the accident." Alternatively, in the financial transaction monitoring video processing scenario within the financial insurance sector, the input video data may include: a monitoring video of a financial trading hall, recording the activities of personnel within the hall during trading hours. The video may contain footage of staff working normally at their respective posts for extended periods, occasionally moving and communicating, as well as repetitive scenes, such as multiple shots of the same trading counter in similar states at different times. The input question data may include: "Find out whether any abnormal personnel entered the trading area and had close contact with staff during a specific trading period."

[0025] In the field of digital healthcare, surgical video processing scenarios may involve input video data that includes: a complete surgical procedure video, encompassing preoperative preparation (repetitive procedures such as disinfection and draping by medical staff), the surgical procedure itself (key actions such as incision and suturing by the surgeon), and postoperative care (repetitive scenes such as wound bandaging and cleaning the surgical site). The preoperative preparation and postoperative care stages contain numerous repetitive actions and scenes. Input question data could include: "Analyzing whether the surgeon's technique in using specific surgical instruments during the procedure was standardized, and the time taken for key surgical steps." Alternatively, in the field of digital healthcare, rehabilitation training video processing scenarios may involve input video data of patients undergoing rehabilitation training, recording their training progress over multiple training cycles. Within each training cycle, patients repeatedly perform the same training movements, such as joint mobilization and muscle stretching, resulting in some overlap in the footage of these movements between cycles. Input question data could include: "Assessing whether the quality of the patient's performance on specific training movements has improved recently, and whether the training intensity is appropriate."

[0026] Step S202: Calculate the similarity of the feature vectors to obtain similarity results, and perform dynamic programming segmentation on the video data based on the similarity results to divide the video data into corresponding video segments.

[0027] In this embodiment, the specific implementation process of calculating the similarity of the feature vectors to obtain the similarity result, and performing dynamic planning segmentation on the video data based on the similarity result to divide the video data into corresponding video segments will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0028] Step S203: Perform time-series merging processing on the video segments to obtain the corresponding video sequence.

[0029] In this embodiment, the specific implementation process of performing time-series merging processing on the video segments to obtain the corresponding video sequence will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0030] Step S204: Compress the video sequence based on a preset token compression strategy to obtain the corresponding visual token sequence.

[0031] In this embodiment, the specific implementation process of compressing the video sequence based on the preset token compression strategy to obtain the corresponding visual token sequence will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0032] Step S205: Based on the attention mechanism, the visual token sequence is evaluated for importance using a preset large language model to obtain the corresponding importance evaluation result.

[0033] In this embodiment, the selection of the aforementioned large language model is not limited and can be determined according to actual business needs. For example, a Video Large Language Model (VideoLLM) can be used. The system calculates the attention score of the visual token sequence in each layer of the deep network used for inference within the selected large language model, which serves as the importance evaluation result. Specifically, the attention score can be calculated using an attention mechanism that assigns a weight to each token based on the relevance between tokens; this weight can then be used as the attention score. For example, in the Transformer model, the attention weight is obtained by calculating the similarity between the query vector, key vector, and value vector, and then the attention score for each token is calculated.

[0034] Step S206: Based on the importance assessment results, the visual token sequence is fused using a preset similarity fusion strategy to obtain the corresponding processed data.

[0035] In this embodiment, the specific implementation process of fusing the visual token sequence using a preset similarity fusion strategy based on the importance assessment results to obtain the corresponding processed data will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0036] Step S207: Based on the large language model, perform reasoning processing on the processed data and the question data to generate corresponding answer data, and output the answer data.

[0037] In this embodiment, the specific implementation process of generating corresponding answer data by reasoning about the processed data and the question data based on the large language model will be further described in detail in subsequent specific embodiments, and will not be elaborated on here. Additionally, the generated answer data can be fed back to the relevant user via email, message, or interface display to complete the output processing of the answer data.

[0038] This application first receives input video data and question data, and extracts features from the video data to obtain corresponding feature vectors. Then, it calculates the similarity of the feature vectors to obtain similarity results, and performs dynamic programming segmentation on the video data based on the similarity results to divide the video data into corresponding video segments. Next, it performs temporal merging on the video segments to obtain corresponding video sequences. Subsequently, it compresses the video sequences based on a preset token compression strategy to obtain corresponding visual token sequences. Based on an attention mechanism, it uses a preset large language model to evaluate the importance of the visual token sequences to obtain corresponding importance evaluation results. Further, based on the importance evaluation results, it uses a preset similarity fusion strategy to fuse the visual token sequences to obtain corresponding processed data. Finally, it performs reasoning processing on the processed data and the question data based on the large language model to generate corresponding answer data, and outputs the answer data. Based on the above automated processing flow, this application adopts a collaborative processing strategy of global planning followed by local refinement. By combining global spatiotemporal compression outside the model with refined information fusion inside the model, it achieves fast and high-precision understanding and question-answering of video data under limited computing resources. This greatly improves the efficiency of video understanding, effectively reduces the computational cost in the video data processing process, increases inference speed, and ensures the accuracy of the generated answer data.

[0039] In some optional implementations, step S202 includes the following steps: performing similarity calculation on the feature vector based on a preset similarity calculation algorithm to obtain the corresponding similarity result.

[0040] In this embodiment, after extracting the feature vector of each frame from the aforementioned video data, a similarity calculation algorithm can be used to calculate the similarity between adjacent frames. This similarity calculation algorithm can employ Euclidean distance, cosine similarity, etc. The similarity can be obtained by normalizing the Euclidean distance or through other transformations; a higher similarity indicates that the content of the two frames is more similar. In this way, a preliminary understanding of the content changes between frames in the video data can be obtained.

[0041] Based on the similarity results, a target segmentation scheme corresponding to the video data is determined using a preset state transition equation.

[0042] In this embodiment, the system does not simply merge two adjacent frames, but uses a dynamic programming algorithm to find the optimal segmentation point globally, dividing the video into several "content-stable" segments. For example, the 10-second scene of the person in red standing at the intersection with minimal changes would be segmented as a single segment. The following state transition equation is used to find the segmentation scheme / optimal segmentation scheme that maximizes the number of pruning steps: The detailed explanations of the formulas and symbols include: : indicates that the first Up to the frame, the video can be pruned (save) to the maximum number of tokens. This is a cumulative optimal value. : Indicates the end position of the previous segment, that is, the position from which the current segment attempts to start from the previous segment. From the beginning of frame to the 1st frame End of frame. The system will iterate through all possible frames. To find the optimal solution. This is a payoff calculation function. It evaluates the payoff if we take the first... Frame to the How much computation can be saved (i.e., how many tokens are clipped) by forcibly merging frames as "the same content"? The benefit calculation function can be designed based on similarity results, specifically determined by the similarity between adjacent frames. The higher the similarity, the more computation is saved after merging. For example, a similarity threshold can be set; when the similarity between adjacent frames is greater than this threshold, they are considered to belong to the same content, and a certain number of tokens can be saved after merging. The number of tokens saved can be estimated based on factors such as the complexity of the frames.

[0043] The video data is segmented based on the target segmentation scheme to obtain corresponding segmented segments.

[0044] In this embodiment, the video data can be effectively optimized, i.e., segmented, based on the obtained target segmentation scheme, and the resulting segmented segments can be used as corresponding video segments. For example, in surveillance video processing, the preceding steps can accurately determine that the 10-second frame of the person in red standing at the intersection shows very little change, thus dividing this part into a segment to achieve the purpose of optimizing video processing.

[0045] The segmented fragment is used as the video segment.

[0046] This application calculates the similarity of feature vectors using a pre-defined similarity calculation algorithm to obtain the corresponding similarity results. Then, based on the similarity results, a pre-defined state transition equation is used to determine the target segmentation scheme corresponding to the video data. Subsequently, the video data is segmented based on the target segmentation scheme to obtain corresponding segmented segments. These segmented segments are then used as video segments. Based on the above processing flow, this application divides video data into video segments by calculating the similarity of feature vectors and using a state transition equation to find a target segmentation scheme. This achieves temporal processing of video data, eliminates duplicate frames, reduces the amount of data processed subsequently, and thus improves the efficiency of video processing and problem-solving.

[0047] In some optional implementations of this embodiment, step S203 includes the following steps: determining the corresponding target segmentation point based on the target segmentation scheme.

[0048] In this embodiment, the segmentation scheme refers to the specific way of dividing the entire video into several segments. It needs to clearly specify the start and end frames of each segment. For example, for a 100-frame video, the segmentation scheme might divide frames 1-20 into the first segment, frames 21-50 into the second segment, and frames 51-100 into the third segment. The target segmentation point (optimal segmentation point) is the key element in determining the segmentation scheme. Specifically, in the dynamic programming segmentation process, for each frame... Through calculation And record To obtain the maximum value Value, this That is the first The optimal split point before the current frame. It represents the optimal split point in the current frame. Previously, the goal was to determine from which frame to begin dividing a new segment to maximize the number of tokens that could be pruned (saved) from the video. Multiple optimal segmentation points must be combined to form a complete segmentation scheme. For example, during the calculation, if the optimal segmentation point for frame 20 is frame 1, the optimal segmentation point for frame 50 is frame 20, and the optimal segmentation point for frame 100 is frame 50, then the segmentation scheme can be determined as frames 1-20, 21-50, and 51-100.

[0049] Based on the target segmentation points, the starting frame and redundant frames in the video segment are identified.

[0050] In this embodiment, based on the optimal segmentation point (i.e., the target segmentation point) calculated by dynamic programming, each video segment (which can be simply referred to as a segment) has a clearly defined start frame and end frame. Within this segment, all subsequent frames except the first frame (start frame) are considered "redundant frames." This is because the start frame already represents the main content of the segment, and subsequent frames have a certain similarity in content to the start frame, resulting in information redundancy. For example, in a surveillance video, a segment depicts a person in red standing at an intersection waiting. The start frame captures the initial state of the person in red standing at the intersection, and the posture and position of the person in red change very little in subsequent frames; these subsequent frames are redundant frames.

[0051] The redundant frames are compressed based on a preset feature compression strategy to obtain the corresponding specified redundant frames.

[0052] In this embodiment, the selection of the above feature compression strategy is not specifically limited, and any of the following can be adopted according to actual business needs: 1) Averaging method: The feature vectors of redundant frames are simply averaged over the feature dimension. Assuming a segment has n redundant frames, and the feature vector dimension of each redundant frame is d, the values ​​at corresponding positions of these n d-dimensional vectors are added together and divided by n to obtain a d-dimensional average feature vector, which is used as the compressed feature of the redundant frames in the segment. For example, if there are 3 redundant frames, and the feature vector dimension of each is 100, the values ​​at corresponding positions are added together and divided by 3 to obtain a 100-dimensional average feature vector. 2) Weighted averaging method: Based on the similarity or other importance indicators between each redundant frame and the starting frame, different weights are assigned to them, and then a weighted average is performed. Redundant frames with higher similarity or greater importance are given greater weights. For example, the weights are determined by calculating the Euclidean distance between the redundant frames and the starting frame; the smaller the distance, the greater the weight. Assuming there are 4 redundant frames with weights of 0.2, 0.3, 0.4, and 0.1, and corresponding feature vectors v1, v2, v3, and v4, the compressed feature vector is 0.2 v1 + 0.3 v2 + 0.4 v3 + 0.1 v4. 3) Dimensionality reduction methods: Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) are used to reduce the dimensionality of the feature vectors of redundant frames. These algorithms can find the most representative feature dimensions in the data, reduce the dimensionality of the feature vectors, and preserve as much information as possible from the original data. For example, PCA can be used to reduce the dimensionality of a 100-dimensional redundant frame feature vector to 50 dimensions.

[0053] The redundant frames can be compressed according to the selected feature compression strategy to obtain the corresponding specified redundant frames.

[0054] The specified redundant frame is attached to the starting frame to obtain the processed specified segment.

[0055] In this embodiment, the compressed redundant frame features are "mounted" onto the first frame (starting frame) of the segment to obtain the corresponding specified segment. This "mounting" is not a simple splicing, but rather a form of association or fusion between the compressed features and the starting frame features, enabling the starting frame to carry the key information of the entire segment. For example, the compressed features can be stored as an additional attribute along with the starting frame features, or the compressed features and starting frame features can be fused (e.g., by performing a weighted average) to obtain a comprehensive feature representation. Thus, in subsequent processing and analysis, only the starting frame and its associated redundant frame features need to be considered to understand the main content of the entire segment. The number of video frames may remain physically unchanged, but logically it has been reorganized into a few key nodes.

[0056] The specified segment is used as the video sequence.

[0057] This application determines the corresponding target segmentation points based on a target segmentation scheme; identifies the start frame and redundant frames in a video segment based on the target segmentation points; then compresses the redundant frames using a preset feature compression strategy to obtain a corresponding specified redundant frame; subsequently, the specified redundant frame is attached to the start frame to obtain the processed specified segment; and finally, the specified segment is used as a video sequence. Based on the above processing flow, this application, by using the target segmentation points corresponding to the target segmentation scheme to identify the start frame and redundant frames in a video segment, and using the feature compression strategy to compress the redundant frames to obtain a specified redundant frame, and then attaching the specified redundant frame to the start frame, and using the resulting processed specified segment as the corresponding video sequence, can efficiently and accurately complete the temporal merging processing of video segments, effectively reducing the amount of data in the obtained video sequence.

[0058] In some optional implementations, step S204 includes the following steps: performing a visual saliency assessment on the target starting frame in the video sequence to obtain the corresponding assessment result.

[0059] In this embodiment, the visual saliency evaluation includes: for the starting frame of each segment in the video sequence, using an attention-based selection mechanism to evaluate the visual saliency of each region in the image. The attention mechanism can simulate the human visual system's focus on important regions in an image, determining its saliency by calculating an attention score for each region. The attention score can be calculated using various methods, such as using a convolutional neural network (CNN) to extract image features, and then calculating the attention weight for each region through fully connected layers as the corresponding evaluation result.

[0060] Based on the evaluation results, the target starting frame is subjected to region selection and information preservation processing to obtain the corresponding first processing information.

[0061] In this embodiment, based on the calculated attention score, regions with high visual saliency, such as the person in red, can be retained while background information, such as the sky and walls, can be directly discarded to obtain the corresponding first-processing information. This highlights key information and reduces unnecessary calculations. For example, for an image containing a person and a background, if the person's attention score is much higher than the background's, then only the person's feature information can be retained, while the background's feature information can be discarded.

[0062] Based on a preset clustering algorithm, the target redundant frames in the video sequence are compressed to obtain the corresponding second processing information.

[0063] In this embodiment, the specific implementation process of compressing information of target redundant frames in the video sequence based on the preset clustering algorithm to obtain the corresponding second processing information will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0064] The first processing information and the second processing information are integrated to obtain the corresponding integrated information.

[0065] In this embodiment, the simplified features of the starting frame (first processing information) obtained from non-redundant token processing and the simplified redundant frame features (second processing information) obtained from redundant token processing can be integrated. During integration, the starting frame features serve as key nodes in the sequence, while the simplified redundant frame features are linked to the starting frame features in some way (such as mounting, associating, etc.), ultimately forming a highly simplified visual token sequence containing key information. This sequence retains the key content of the video while significantly reducing the amount of data, providing an efficient foundation for subsequent processing. For example, in surveillance video processing, the final visual token sequence may only contain features of a few key frames and their associated simplified redundant frame features, accurately representing the main information of the entire video segment.

[0066] The integrated information is used as the visual token sequence.

[0067] This application evaluates the visual saliency of the target start frame in a video sequence to obtain the corresponding evaluation result. Based on the evaluation result, it performs region selection and information preservation processing on the target start frame to obtain the corresponding first processing information. Then, it compresses the target redundant frames in the video sequence using a preset clustering algorithm to obtain the corresponding second processing information. The first and second processing information are then integrated to obtain the integrated information, which is subsequently used as a visual token sequence. Based on the above processing flow, this application employs different compression strategies for tokens from different sources (start frame and redundant frames). For the start frame, regions with high visual saliency are retained to highlight key information; for redundant frames, a clustering algorithm is used to extract representative information and discard edge noise, thereby obtaining an extremely concise visual token sequence. This effectively reduces the data volume of the visual token sequence, which is beneficial for providing efficient data support for large language models.

[0068] In some optional implementations, the information compression processing of the target redundant frames in the video sequence based on the preset clustering algorithm to obtain the corresponding second processing information includes the following steps: calling a density-based clustering algorithm.

[0069] In this embodiment, the density-based clustering algorithm is specifically the DPC-KNN algorithm, which includes: .

[0070] Wherein, the token / visual token is the vector representation of a video frame after encoder processing. Detailed explanations of the formulas and symbols include: : Represents a token The local density score indicates that the token is surrounded by more similar features, making it more representative. : Represents the distance to the token in the feature space. The k nearest neighbor tokens. The system calculates the Euclidean distance between two tokens, representing their difference. Using this formula, the system identifies and retains the cluster centers with the highest density, discarding edge noise. This is analogous to selecting only a few key snapshots that best represent changes in motion from a set of similar photos.

[0071] The density-based clustering algorithm is used to calculate and process the target redundant frames in the video sequence to obtain the corresponding local density scores.

[0072] In this embodiment, the target redundant frames in the above video sequence can be calculated and processed using the density-based clustering algorithm described above to obtain local density scores.

[0073] Based on the local density score, cluster center selection is performed on the target redundant frame to obtain the corresponding designated frame.

[0074] In this embodiment, the selection of cluster centers includes selecting and retaining the cluster centers with the highest density based on the calculated local density scores. These cluster centers represent key information in the redundant frames and can effectively summarize the characteristics of the redundant frames. For example, for a group of similar redundant frames, a clustering algorithm can find several of the most representative frames, retain them as cluster centers, and discard other edge noise frames.

[0075] The specified frame is fused based on a preset fusion strategy to obtain the corresponding fusion result.

[0076] In this embodiment, the information of the retained cluster centers (specified frames) can be fused to obtain a simplified feature representation of the redundant frames. The fusion method can employ averaging, weighted averaging, or other more complex methods, depending on the specific application requirements. Complex fusion methods may include using complex models such as neural networks for information fusion. Neural networks can learn the nonlinear relationships between features of different cluster centers and automatically adjust the fusion method through training to obtain a better feature representation. Example: Using a simple multilayer perceptron (MLP) for fusion. The cluster center feature vectors are concatenated as input, processed through multiple fully connected layers and activation functions, and finally output as a fused feature vector. For example, with 3 cluster centers, each feature vector having a dimension of 50, they are concatenated into a 150-dimensional vector and input into the MLP. After passing through two fully connected layers (100 neurons per layer, using the ReLU activation function), a 50-dimensional fused feature vector is output. Furthermore, after fusion, the features can be further compressed to reduce the data volume. Common methods include principal component analysis (PCA) and feature selection. For example, PCA can be used to reduce the dimensionality of the fused feature vectors, retaining the most important principal components, thus obtaining a more concise feature representation.

[0077] The fusion result is used as the second processing information.

[0078] This application employs a density-based clustering algorithm to calculate and process target redundant frames in a video sequence, obtaining corresponding local density scores. Then, based on these local density scores, cluster centers are selected from the target redundant frames to obtain designated frames. Subsequently, these designated frames are fused using a preset fusion strategy to obtain the fusion result. This fusion result is then used as the second processing information. Based on this processing flow, this application extracts representative key information from redundant frames through a comprehensive process of calculating local density scores, selecting cluster centers, and fusing information. This allows for the efficient and accurate generation of concise feature representations of redundant frames, improving the accuracy of the generated second processing information.

[0079] In some optional implementations of this embodiment, step S206 includes the following steps: identifying the primary and secondary tokens in the visual token sequence based on the importance assessment results.

[0080] In this embodiment, based on the calculated importance assessment results, i.e., the attention score, it is possible to identify which of the above visual token sequences are "primary tokens" (high scores) and which are "secondary tokens" (low scores). Primary tokens typically contain key information from the video and have a significant impact on the model's inference results; secondary tokens contain some relatively less important information, but this information may still provide some assistance to the model's inference.

[0081] In related models (such as those based on the Transformer architecture), input data (such as text and images) is segmented into small units, which are called tokens. For example, when processing text, a word or sub-word might be a token; when processing images, the feature vectors of local regions segmented from the image can also be considered tokens. Specifically, during model execution, an attention score is calculated for each token. The attention score reflects the importance of that token in the model's processing. Based on the calculated attention scores, tokens with high scores are identified as "primary tokens," which typically contain key and core information from the input data; while tokens with low scores are identified as "secondary tokens," which carry relatively less important information and have a smaller impact on the model's final output.

[0082] The secondary token is subjected to similarity matching with the primary token to determine a similar token corresponding to the secondary token from the primary token.

[0083] In this embodiment, the similarity matching process includes: for each low-scoring token (i.e., secondary token) to be pruned, finding its most similar "relative" primary token (i.e., similar token). Similarity can be determined by calculating metrics such as Euclidean distance or cosine similarity between the feature vectors of two tokens. For example, the similarity between the low-scoring token and all primary tokens is calculated, and the primary token with the highest similarity is selected as its "relative," i.e., similar token.

[0084] Obtain the preset information fusion formula.

[0085] In this embodiment, the information fusion formula includes: The detailed explanations of the formulas and symbols include: : The updated reserved token feature vector, which incorporates information from other tokens. Tokens that were originally deemed important and needed to be retained. This is a group of low-scoring tokens that are about to be pruned. Instead of deleting them directly, the system finds their most similar "relatives". Average their information and incorporate it into This step ensures that even seemingly unimportant information is retained in the main token in a "subconscious" manner for use in subsequent reasoning.

[0086] Based on the information fusion formula, the primary token, the secondary token, and the similar token are subjected to information fusion processing to obtain the corresponding fused data.

[0087] In this embodiment, the primary token, secondary token, and similar tokens can be fused based on the above information fusion formula, and the resulting fused data can be used as the corresponding processing data.

[0088] Among these, the secondary tokens identified earlier, due to their low attention scores and low importance, contribute limitedly to the model output and are therefore considered objects that can be optimized, i.e., "low-scoring tokens about to be pruned". In the similarity fusion step, the most similar "relative" primary token is found for these low-scoring tokens about to be pruned, and then the information of the low-scoring tokens is fused into the primary tokens, instead of being discarded directly. This reduces the number of tokens while preserving as much information as possible from the input data.

[0089] In addition, the processed data is a high-quality key-value (KV) cache (short-term memory) that has undergone multiple rounds of filtering and fusion. This KV cache will serve as input for the next step of processing, providing accurate visual memory support for generating the final answer.

[0090] The fused data is used as the processed data.

[0091] In this embodiment, the goal of the processing steps is to further optimize memory usage and prevent information loss in deep networks performing reasoning on large language models. Traditional methods directly delete low-scoring tokens, resulting in the loss of subtle clues. However, this application identifies primary and secondary tokens by calculating their attention scores, and then performs a fusion operation to integrate the information from low-scoring tokens into the primary tokens, ensuring that key information is retained "subconsciously" for subsequent reasoning.

[0092] This application identifies primary and secondary tokens in a visual token sequence based on importance assessment results. Then, it performs similarity matching on the secondary and primary tokens to determine similar tokens from the primary tokens. Next, it obtains a preset information fusion formula and performs information fusion processing on the primary, secondary, and similar tokens based on this formula to obtain corresponding fused data. This fused data is then used as processing data. Based on this process, this application identifies primary and secondary tokens by calculating the attention score of the visual token sequence, determines similar tokens based on similarity matching, and then performs a fusion operation based on the information fusion formula to integrate information from low-scoring tokens into the primary tokens. This ensures that key information is retained subconsciously, effectively improving the accuracy and adaptability of the generated processing data, and providing accurate visual memory support for generating the final answer.

[0093] In some optional implementations of this embodiment, step S207, which involves performing reasoning processing on the processed data and the question data based on the large language model to generate corresponding answer data, includes the following steps: based on preset reasoning rules, using the decoder in the large language model to perform reasoning and judgment processing on the processed data and the question data to obtain corresponding reasoning results.

[0094] In this embodiment, the above reasoning and judgment processing includes: 1. Input preparation. KV cache: This is an intermediate result generated by the large language model when processing video features. During video processing, the model encodes the features of each frame and stores the key and value information in the encoding process to form a KV cache. These caches contain key information of video features at different processing stages, which helps the model quickly review and locate relevant content. Question data: The question raised by the user is a query requirement for video content, such as "Where did the person in red last appear in the video?" The question is converted into a semantic representation that the model can understand, usually encoded into a vector by a text encoder. This vector contains the key semantic information of the question. Highly condensed visual memory: This is the video feature representation obtained after the previous steps. Through dynamic programming segmentation and temporal merging, the video is divided into several segments. The redundant frame features of each segment are compressed and attached to the starting frame, thus forming a highly condensed visual memory. These memories are stored in the form of feature vectors, and each feature vector represents the key information of a segment.

[0095] 2. Localization Process. Similarity Calculation: The large language model calculates the similarity between the semantic vector of the user's question and the key information in the KV cache. Various methods can be used for similarity calculation, such as cosine similarity. For each key vector, its cosine similarity to the question's semantic vector is calculated. The higher the similarity, the greater the relevance between the video features corresponding to that key vector and the user's question. Key Feature Localization: Based on the similarity calculation results, the model selects the values ​​corresponding to the key vectors with the highest similarity. These values ​​contain the video feature segments most relevant to the user's question. Simultaneously, the model can combine feature vectors from highly condensed visual memory to further filter out key features directly related to the question. For example, if the user's question is about the last location of the person in red, the model will prioritize video segments containing the feature of the person in red and appearing later in time.

[0096] 3. Feature Fusion. Multi-source Feature Integration: The large language model fuses the located key features (including values ​​retrieved from the KV cache and relevant features from highly condensed visual memory). Fusion can be a simple concatenation or a weighted concatenation based on feature importance. For example, if a feature is more relevant to the user's question, it can be given a larger weight, making it more prominent in the fused features. Contextual Information Integration: In addition to the key features themselves, the model also considers the contextual information of the video. For example, the scene information before and after the appearance of the person in red in the video may affect the judgment of their last appearance location. The model combines this contextual information with the key features to form a more comprehensive feature representation.

[0097] 4. Reasoning and Judgment. Rule-based reasoning: Large language models use decoders to reason and judge based on predefined rules of the fused features. For example, if the last scene of the person in red in the video is at a specific location with obvious landmarks (such as a red mailbox), the model can determine that the person in red last appeared near that mailbox based on these rules. Statistical reasoning: Large language models can also use statistical methods for reasoning. For example, by statistically analyzing the frequency of the person in red appearing in different locations in the video, the location with the highest frequency is likely closer to their last appearance. Alternatively, predictions can be made based on the person in red's movement trajectory, combining their direction and speed to infer their likely last location.

[0098] Based on the reasoning results, text planning processing is performed to obtain the corresponding text planning results.

[0099] In this embodiment, the text planning process includes: Semantic structure determination: Based on the reasoning results, the model determines the semantic structure of the answer. For example, for the question "Where did the person in red last appear in the video?", the semantic structure of the answer might include "The person in red last appeared at [specific location]". Information organization: The model organizes the key information obtained through reasoning, determining which information needs to be highlighted in the answer and which information can be used as supplementary explanations. For example, if in addition to the specific location, there is information such as the time of the person in red's appearance and the surrounding environment, the model will determine the presentation order and method of this information based on the focus of the question and the user's potential needs.

[0100] Based on the text planning results, the answer generation process is performed to obtain the corresponding generated text.

[0101] In this embodiment, the above-mentioned answer generation process includes: Lexical selection: The large language model selects appropriate words from the vocabulary database to construct the answer based on semantic structure and organized information. Lexical selection considers the semantic accuracy, fluency, and coherence with the context. For example, for location descriptions, accurate location words such as "intersection" and "next to the mailbox" are selected. Sentence construction: The selected words are combined into sentences according to grammatical rules. The model ensures that the sentences are grammatically correct, clear in meaning, and conform to the expression habits of natural language. For example, the generated sentence might be "The person in red last appeared next to the red mailbox at the intersection."

[0102] The generated text is used as the answer data.

[0103] This application uses a decoder in a large language model to perform reasoning and judgment processing on the processed data and question data based on preset reasoning rules, obtaining corresponding reasoning results. Then, based on the reasoning results, text planning processing is performed to obtain corresponding text planning results. Subsequently, based on the text planning results, answer generation processing is performed to obtain corresponding generated text. This generated text is then used as the answer data. Based on the above processing flow, this application, through a decoder based on a large language model, utilizes processed information, combined with question data, and leverages highly condensed visual memory for reasoning. Because interfering information is removed and key clues are retained, the large language model can quickly locate key information in the video data, thereby generating accurate answer data.

[0104] In some alternative implementations, the user information obtained is subject to user consent and complies with relevant laws and policies.

[0105] Furthermore, any software tools or components not belonging to our company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0106] Furthermore, the core objective of this application is to provide a training-free, plug-and-play overall token merging framework. This method aims to address the problems of "fragmentation of internal and external optimization" and "loss of global temporal information" mentioned in the background art. By coordinating external spatiotemporal compression with internal similarity fusion, it significantly reduces computational costs and improves inference speed.

[0107] The core innovations of this application are as follows: 1. A globally redundant-aware temporal segmentation mechanism: Unlike traditional methods that only consider a few adjacent frames, this invention introduces a global optimization strategy based on dynamic programming. Like an editor, it examines the entire video timeline, intelligently identifying segments with long-term unchanged content (such as long static shots), and merging the tokens in these segments to maximize the elimination of temporal redundancy.

[0108] 2. Spatial Merging Mechanism of Hybrid Strategy: In the spatial dimension, this invention employs differentiated processing based on the source of the tokens. For non-redundant tokens within a single frame, an attention mechanism is used to preserve key regions; for redundant tokens generated by temporal merging, a density-based clustering algorithm is creatively introduced to ensure that the compressed information not only retains visual features but also temporal representativeness.

[0109] 3. Internal Similarity Fusion Mechanism: This is an innovation of this invention within the model. Traditional internal pruning often directly discards unimportant tokens, resulting in permanent information loss. This invention uses "fusion" instead of "discarding," "blending" the information of tokens deemed unimportant into the most similar retained tokens. This reduces computation while preserving semantic information to the greatest extent possible.

[0110] Through the above innovations, this application can reduce the computational load to less than 10% of the original while maintaining more than 99% of the original model performance, thus greatly improving the efficiency of long video understanding.

[0111] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0112] It should be emphasized that, to further ensure the privacy and security of the above-mentioned answer data, the answer data can also be stored in a blockchain node.

[0113] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0114] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0116] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0117] Referring further to FIG3, as an implementation of the method shown in FIG2, this application provides an embodiment of an artificial intelligence-based data processing device, which corresponds to the method embodiment shown in FIG2, and the device can be specifically applied to various electronic devices.

[0118] As shown in Figure 3, the artificial intelligence-based data processing device 300 described in this embodiment includes: a first processing module 301, a second processing module 302, a merging module 303, a compression module 304, an evaluation module 305, a fusion module 306, and a third processing module 307. The system comprises: a first processing module 301, used to receive input video data and question data, and extract features from the video data to obtain corresponding feature vectors; a second processing module 302, used to calculate the similarity of the feature vectors to obtain similarity results, and to perform dynamic programming segmentation on the video data based on the similarity results to divide the video data into corresponding video segments; a merging module 303, used to perform temporal merging on the video segments to obtain corresponding video sequences; a compression module 304, used to compress the video sequences based on a preset token compression strategy to obtain corresponding visual token sequences; an evaluation module 305, used to evaluate the importance of the visual token sequences using a preset large language model based on an attention mechanism to obtain corresponding importance evaluation results; a fusion module 306, used to perform fusion operations on the visual token sequences using a preset similarity fusion strategy based on the importance evaluation results to obtain corresponding processed data; and a third processing module 307, used to perform reasoning processing on the processed data and question data based on the large language model to generate corresponding answer data, and to output the answer data.

[0119] In some optional implementations of this embodiment, the second processing module 302 includes: a calculation submodule, used to perform similarity calculation on the feature vector based on a preset similarity calculation algorithm to obtain a corresponding similarity result; a first determination submodule, used to determine a target segmentation scheme corresponding to the video data based on the similarity result using a preset state transition equation; a segmentation submodule, used to perform segmentation processing on the video data based on the target segmentation scheme to obtain a corresponding segmented segment; and a second determination submodule, used to use the segmented segment as the video segment.

[0120] In some optional implementations of this embodiment, the merging module 303 includes: a third determining submodule, used to determine the corresponding target segmentation point based on the target segmentation scheme; a first identification submodule, used to identify the start frame and redundant frames in the video segment based on the target segmentation point; a first compression submodule, used to compress the redundant frames based on a preset feature compression strategy to obtain the corresponding specified redundant frame; an mounting submodule, used to mount the specified redundant frame to the start frame to obtain the processed specified segment; and a fourth determining submodule, used to use the specified segment as the video sequence.

[0121] In some optional implementations of this embodiment, the compression module 304 includes: an evaluation submodule, used to perform visual saliency evaluation on the target starting frame in the video sequence to obtain a corresponding evaluation result; a processing submodule, used to perform region selection and information preservation processing on the target starting frame based on the evaluation result to obtain corresponding first processing information; a second compression submodule, used to perform information compression processing on the target redundant frames in the video sequence based on a preset clustering algorithm to obtain corresponding second processing information; an integration submodule, used to integrate the first processing information and the second processing information to obtain corresponding integrated information; and a fifth determination submodule, used to use the integrated information as the visual token sequence.

[0122] In some optional implementations of this embodiment, the second compression submodule includes: a calling unit for calling a density-based clustering algorithm; a calculation unit for calculating and processing target redundant frames in the video sequence based on the density-based clustering algorithm to obtain corresponding local density scores; a selection unit for performing cluster center selection processing on the target redundant frames based on the local density scores to obtain corresponding specified frames; a fusion unit for performing fusion processing on the specified frames based on a preset fusion strategy to obtain corresponding fusion results; and a determination unit for using the fusion results as the second processing information.

[0123] In some optional implementations of this embodiment, the fusion module 306 includes: a second identification submodule, used to identify the primary and secondary tokens in the visual token sequence based on the importance assessment result; a sixth determination submodule, used to perform similarity matching processing on the secondary tokens and the primary tokens to determine similar tokens corresponding to the secondary tokens from the primary tokens; an acquisition submodule, used to acquire a preset information fusion formula; a fusion submodule, used to perform information fusion processing on the primary tokens, the secondary tokens, and the similar tokens based on the information fusion formula to obtain corresponding fused data; and a seventh determination submodule, used to use the fused data as the processed data.

[0124] In some optional implementations of this embodiment, the third processing module 307 includes: a reasoning submodule, used to perform reasoning and judgment processing on the processed data and the question data using the decoder in the large language model based on preset reasoning rules, to obtain the corresponding reasoning result; a planning submodule, used to perform text planning processing based on the reasoning result, to obtain the corresponding text planning result; a generation submodule, used to perform answer generation processing based on the text planning result, to obtain the corresponding generated text; and an eighth determining submodule, used to use the generated text as the answer data.

[0125] To address the aforementioned technical problems, this application also provides a computer device. Please refer to Figure 4 for details; Figure 4 is a basic structural block diagram of the computer device according to this embodiment.

[0126] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0127] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0128] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for data processing methods based on artificial intelligence. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0129] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the artificial intelligence-based data processing method.

[0130] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0131] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the artificial intelligence-based data processing method described above.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0133] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A data processing method based on artificial intelligence, characterized in that, The process includes the following steps: receiving input video data and question data, extracting features from the video data to obtain corresponding feature vectors; calculating the similarity of the feature vectors to obtain similarity results, and performing dynamic programming segmentation on the video data based on the similarity results to divide the video data into corresponding video segments. The video segments are time-series merged to obtain the corresponding video sequence; The video sequence is compressed based on a preset token compression strategy to obtain the corresponding visual token sequence. Based on the attention mechanism, the visual token sequence is evaluated using a pre-defined large language model to obtain the corresponding importance evaluation results; Based on the importance assessment results, the visual token sequence is fused using a preset similarity fusion strategy to obtain the corresponding processed data. Based on the large language model, the processed data and the question data are subjected to reasoning processing to generate corresponding answer data, and the answer data is then output.

2. The data processing method based on artificial intelligence according to claim 1, characterized in that, The step of calculating the similarity of the feature vectors to obtain a similarity result, and performing dynamic programming segmentation on the video data based on the similarity result to divide the video data into corresponding video segments, specifically includes: calculating the similarity of the feature vectors based on a preset similarity calculation algorithm to obtain a corresponding similarity result; determining a target segmentation scheme corresponding to the video data using a preset state transition equation based on the similarity result; segmenting the video data based on the target segmentation scheme to obtain corresponding segmented segments; and using the segmented segments as the video segments.

3. The data processing method based on artificial intelligence according to claim 2, characterized in that, The step of performing temporal merging processing on the video segments to obtain the corresponding video sequence specifically includes: determining the corresponding target segmentation point based on the target segmentation scheme; identifying the start frame and redundant frames in the video segment based on the target segmentation point; compressing the redundant frames based on a preset feature compression strategy to obtain the corresponding specified redundant frame; attaching the specified redundant frame to the start frame to obtain the processed specified segment; and using the specified segment as the video sequence.

4. The data processing method based on artificial intelligence according to claim 1, characterized in that, The step of compressing the video sequence based on a preset token compression strategy to obtain a corresponding visual token sequence specifically includes: performing visual saliency evaluation on the target start frame in the video sequence to obtain a corresponding evaluation result; performing region selection and information preservation processing on the target start frame based on the evaluation result to obtain corresponding first processing information; performing information compression processing on the target redundant frames in the video sequence based on a preset clustering algorithm to obtain corresponding second processing information; integrating the first processing information and the second processing information to obtain corresponding integrated information; and using the integrated information as the visual token sequence.

5. The data processing method based on artificial intelligence according to claim 4, characterized in that, The step of compressing information of target redundant frames in the video sequence based on a preset clustering algorithm to obtain corresponding second processing information specifically includes: calling a density-based clustering algorithm; calculating and processing the target redundant frames in the video sequence based on the density-based clustering algorithm to obtain corresponding local density scores; selecting cluster centers for the target redundant frames based on the local density scores to obtain corresponding designated frames; performing fusion processing on the designated frames based on a preset fusion strategy to obtain corresponding fusion results; and using the fusion results as the second processing information.

6. The data processing method based on artificial intelligence according to claim 1, characterized in that, The step of fusing the visual token sequence using a preset similarity fusion strategy based on the importance assessment result to obtain corresponding processed data specifically includes: identifying the primary and secondary tokens in the visual token sequence based on the importance assessment result; performing similarity matching processing on the secondary tokens and the primary tokens to determine similar tokens corresponding to the secondary tokens from the primary tokens; obtaining a preset information fusion formula; performing information fusion processing on the primary tokens, the secondary tokens, and the similar tokens based on the information fusion formula to obtain corresponding fused data; and using the fused data as the processed data.

7. The data processing method based on artificial intelligence according to claim 1, characterized in that, The step of generating corresponding answer data by reasoning about the processed data and the question data based on the large language model specifically includes: using a decoder in the large language model to perform reasoning and judgment processing on the processed data and the question data based on preset reasoning rules to obtain corresponding reasoning results; performing text planning processing based on the reasoning results to obtain corresponding text planning results; performing answer generation processing based on the text planning results to obtain corresponding generated text; and using the generated text as the answer data.

8. A data processing device based on artificial intelligence, characterized in that, include: The first processing module is used to receive input video data and problem data, and to extract features from the video data to obtain corresponding feature vectors. The second processing module is used to calculate the similarity of the feature vector to obtain the similarity result, and to perform dynamic programming segmentation processing on the video data based on the similarity result to divide the video data into corresponding video segments. The merging module is used to perform time-series merging processing on the video segments to obtain the corresponding video sequence; The compression module is used to compress the video sequence based on a preset token compression strategy to obtain the corresponding visual token sequence. The evaluation module is used to evaluate the importance of the visual token sequence based on the attention mechanism and using a preset large language model to obtain the corresponding importance evaluation results; The fusion module is used to perform a fusion operation on the visual token sequence based on the importance assessment results and using a preset similarity fusion strategy to obtain the corresponding processed data. The third processing module is used to perform reasoning processing on the processed data and the question data based on the large language model to generate corresponding answer data, and to output the answer data.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the data processing method based on artificial intelligence as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data processing method based on artificial intelligence as described in any one of claims 1 to 7.

Images