Large model-based material playing method and large model training method

Abstract

The invention provides a large model-based material playing method and a large model training method, and relates to the technical field of intelligent terminals, the method comprises the steps of obtaining a to-be-played material, calling a trained large model to identify target content in the to-be-played material in response to a received annotation instruction for the to-be-played material, and playing the to-be-played material according to the target content. Obtaining a first position and labeling mode information of the target content; determining a pattern type used for identifying the first position based on labeling mode information of target content; generating annotation data corresponding to the target content in the to-be-played material according to the pattern type and the first position; and controlling the terminal equipment to play the to-be-played material including the annotation data. According to the method, the target content included in the to-be-played material identified by the large model is labeled, the labeled data labeled according to the pattern type at the first position is obtained, and the terminal equipment is controlled to play the to-be-played material including the labeled data, so that a user can timely obtain important content in the material played by the terminal equipment.

Description

Technical Field

[0001] This application relates to the field of smart terminal technology, and in particular to a method for playing media based on a large model and a method for training a large model. Background Technology

[0002] Screen terminals can be used to display real-time and important information. For example, they can be set up in various environments such as airports, bus stops, train stations, and hospitals to display materials such as Word documents and PowerPoint presentations. However, with existing screen terminal display solutions, when the amount of data to be displayed is large, users cannot quickly access the key content they want to focus on. Summary of the Invention

[0003] This application provides a method for playing back media based on a large model and a method for training a large model, in order to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, embodiments of this application provide a method for playing media based on a large model. The method includes: acquiring media to be played; responding to receiving a labeling instruction for the media to be played, calling a trained large model to identify target content in the media to be played, and obtaining a first position and labeling method information of the target content; determining a pattern type for identifying the first position based on the labeling method information of the target content; generating labeling data corresponding to the target content in the media to be played according to the pattern type and the first position; and controlling a terminal device to play the media to be played including the labeling data.

[0005] Secondly, embodiments of this application provide a large model training method, including: acquiring first playback material training data including pattern label data; wherein the pattern label data is used to identify target content in the first playback material training data; training a preset large model using the first playback material training data to obtain a trained large model, and implementing the above method based on the trained large model.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-mentioned embodiments.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0008] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a computer program, which, when executed by a processor, implements the method described in any of the above-mentioned embodiments.

[0009] Compared with related technologies, this application has the following advantages:

[0010] This application provides a method for playing media based on a large model, including: acquiring media to be played; responding to receiving an annotation instruction for the media to be played, calling a trained large model to identify target content in the media to be played, and obtaining a first position and annotation method information of the target content; determining a pattern type for identifying the first position based on the annotation method information of the target content; generating annotation data corresponding to the target content in the media to be played according to the pattern type and the first position; and controlling a terminal device to play the media to be played including the annotation data. According to this application embodiment, the target content included in the media to be played identified by the large model can be annotated, thereby obtaining annotation data annotated at the first position according to the pattern type. Finally, the terminal device is controlled to play the media to be played including the annotation data, so that the user can promptly obtain important content from the media played by the terminal device.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0013] Figure 1 A flowchart of a material playback method based on a large model provided in an embodiment of this application is shown;

[0014] Figure 2 This illustration shows an architecture diagram of a terminal device that can intelligently highlight key points and answer user questions when playing programs, according to an embodiment of this application.

[0015] Figure 3 A schematic diagram of a large model algorithm provided in an embodiment of this application is shown;

[0016] Figure 4 This illustration shows a schematic diagram of a key highlighting algorithm provided in an embodiment of this application;

[0017] Figure 5 A schematic diagram of a large model training method provided in an embodiment of this application is shown;

[0018] Figure 6 A block diagram of an electronic device used to implement embodiments of this application is shown. Detailed Implementation

[0019] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0020] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0021] The existing technical solution for playing programs on terminal screens involves users uploading materials such as Word documents and PPT presentations to a cloud-based information publishing system, followed by program arrangement: the uploaded materials are laid out, borders are set, and background colors are filled in the program editor. The next step uses the MQTT protocol (Message Queuing Telemetry Transport, a lightweight messaging protocol) to send instructions to the terminal device. Upon receiving the instructions, the terminal device calls the information publishing system interface to retrieve the playback content and plays it.

[0022] Currently, most content layout is done manually, requiring people to understand the material and then format it into PPT slides, images, and videos, without allowing for interaction. Furthermore, users cannot immediately access support when they have questions, which is a significant drawback for product promotion. Additionally, the large playback area may prevent users from immediately grasping the key points.

[0023] Based on this, this application provides a method for playing back materials using a large model and a method for training a large model, which can intelligently highlight key points in the played materials using a large model and a key highlighting algorithm. Simultaneously, when a user is detected watching the program, the system proactively initiates voice interaction to answer questions raised by the user within the scope of the played program content.

[0024] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0025] This application provides a method for playing media based on a large model. This method can be applied to a server, where the server and terminal device communicate with each other. Figure 1 The diagram shown is a flowchart of a large-model-based media playback method according to an embodiment of this application. The method may include:

[0026] Step S101: Obtain the material to be played. In response to receiving the annotation instruction for the material to be played, call the trained large model to identify the target content in the material to be played, and obtain the first position and annotation method information of the target content.

[0027] In this embodiment, the material to be played is a file used for display on a terminal device, such as, but not limited to, video files, Word files, PowerPoint files, and other forms of files including text and / or image information. The annotation instructions for the material to be played are used to trigger a highlighting operation, that is, to call a trained large model to identify target content in the material to be played. The target content is the key content in the material to be played, such as, but not limited to, the title, core concepts and definitions, advantages analysis, disadvantage analysis, usage scenarios, tools and resources, summary and conclusions, etc.

[0028] It should be noted that when users upload materials to be played to the server, they can select or unselect the highlighting function. When the user selects the highlighting function, the user can send annotation instructions for the materials to be played to the server through the selection operation.

[0029] In this embodiment, the large model can be a Natural Language Processing (NLP) model, such as, but not limited to, GPT series models, BERT series models, or T5, etc. The large model can be pre-trained to enable it to circle and highlight target content in the playback material. Because the large model has this ability, its output can include the position of the target content, i.e., the first position, and annotation information. The first position can be used to determine the relative position of the target content within the playback material. For example, if the playback material is a Word document, the first position can be used to determine the page number, line number, and text number of the target content within that document. The annotation information can include a highlighting token for the target content. For example, in one possible implementation, the output of the large model includes: the position of the highlighted content and the highlighting token, wherein the highlighting token is used for highlighting and highlighting the target content. <oval>(x,y,r) represents drawing an ellipse; <line>(x_start, y_start, length) represents drawing a line; <text>(text, x, y) represents drawing text. It's important to note that in Natural Language Processing (NLP), a "token" typically refers to a basic unit of text. This unit can be a word, punctuation mark, or other symbol. For example, the sentence "Hello, world!" can be segmented into the following tokens: "Hello", , ", world", and "!".

[0030] In this embodiment of the application, the key content in the material to be played can be intelligently identified using a large model algorithm.

[0031] Step S102: Determine the pattern type used to identify the first position based on the annotation method information of the target content.

[0032] In this embodiment of the invention, based on one or more data included in the annotation method information of the target content, it is possible to determine how to identify the target content, that is, to determine the pattern type used to identify the first position based on the annotation method information of the target content. The pattern type includes the pattern shape, the pattern size, and the relative positional relationship between the pattern shape and the first position. For example, the pattern shape used to mark the target content can be determined to be: a circle, an ellipse, a rectangle, a star, an irregular closed shape, a curve, a line segment, and / or text, etc. The pattern size can also be determined based on the annotation method information of the target content; for example, multiple key coordinate points of the pattern can be determined, and these key coordinate points are used to determine the size of the display area required by the pattern.

[0033] In this embodiment, the annotation information obtained in step S101, such as the obtained circle / draw token, is converted into a corresponding pattern type using a pattern conversion API (Application Programming Interface). In this step, pattern conversion components corresponding to different pattern types can be pre-set. These components are used to convert the circle / draw token into a specified pattern. The API of the pattern conversion component is called to determine the pattern type used to identify the first position based on the annotation information of the target content.

[0034] Step S103: Generate annotation data corresponding to the target content in the material to be played according to the pattern type and the first position.

[0035] In this embodiment of the invention, a drawing component is pre-configured; for example, the drawing component can be Apache POI (Poor Obfuscation Implementation). The drawing component is used to draw specified content onto a specified file. The API of the drawing component is called to generate annotation data corresponding to the target content in the material to be played, according to the pattern type and the first position.

[0036] In one possible implementation, if the presentation material is a PowerPoint file, Apache POI can be used to highlight key content within the PowerPoint slides, and then the highlighted slides can be saved again. It should be noted that Apache POI is an open-source Java library for reading and writing Microsoft Office documents. It supports processing Microsoft Office format files, including Microsoft Word, Excel, and PowerPoint.

[0037] Step S104: Control the terminal device to play the material to be played, which includes the labeled data.

[0038] In this embodiment of the invention, the material played by the terminal device can be intelligently highlighted, allowing users to focus on key parts of the material. It should be noted that the terminal device is equipped with a built-in camera and microphone, enabling interaction with the user. Communication between the terminal device and the server can be based on the MQTT protocol.

[0039] This application provides a method for playing media based on a large model, including: acquiring media to be played; responding to receiving an annotation instruction for the media to be played, calling a trained large model to identify target content in the media to be played, and obtaining a first position and annotation method information of the target content; determining a pattern type for identifying the first position based on the annotation method information of the target content; generating annotation data corresponding to the target content in the media to be played according to the pattern type and the first position; and controlling a terminal device to play the media to be played including the annotation data. According to this application embodiment, the target content included in the media to be played identified by the large model can be annotated, thereby obtaining annotation data annotated at the first position according to the pattern type. Finally, the terminal device is controlled to play the media to be played including the annotation data, so that the user can promptly obtain important content from the media played by the terminal device.

[0040] In one possible implementation, determining the pattern type for identifying the first position based on the annotation method information of the target content can be performed by the following steps: determining display effect data and a second position of the pattern type based on the annotation method information of the target content; and determining the pattern type for identifying the first position of the target content according to the display effect data and the second position.

[0041] In this embodiment, the display effect of the pattern type includes, but is not limited to, the pattern color and the stroke width of the pattern. The second position is used to determine the area size and / or coordinates required by the pattern type.

[0042] In one possible implementation, the pattern type is a line segment; determining the display effect data and second position of the pattern type based on the annotation method information of the target content includes: determining the stroke width of the line segment according to the annotation method information of the target content to obtain the display effect data of the pattern type, and determining the endpoints of the line segment to obtain the coordinate data corresponding to the second position.

[0043] In this possible implementation, the display effect data for the pattern type includes, but is not limited to, the width of the line segment. The coordinate data corresponding to the second position includes, but is not limited to, the endpoints of the line segment. When the pattern type is a line segment, the algorithm for converting a circled token to a pattern can be referenced below:

[0044] private static void drawLine(double x_start,double y_start,length){

[0045] Graphics2D g2d=(Graphics2D)Graphics.create();

[0046] g2d.setRenderingHint(RenderingHints.KEY_ANTIALIASING,RenderingHints.VALUE_ANTIALIAS_ON);

[0047] g2d.setColor(Color.RED);

[0048] / / Draw a line segment between two points (set line width to 5px): point (50, 150), point (200, 150)

[0049] double x_end=x_start+length;

[0050] double y_end=x_start+length;

[0051] / / Stroke outline (pen width / line width is 5px)

[0052] BasicStroke bs1=new BasicStroke(5);

[0053] g2d.setStroke(bs1);

[0054] g2d.drawLine(x_start,y_start,200,150);

[0055] g2d.dispose();

[0056] }

[0057] In one possible implementation, the pattern type is an ellipse; determining the display effect data and second position of the pattern type based on the annotation method information of the target content includes: determining the fill data of the ellipse according to the annotation method information of the target content to obtain the display effect data of the pattern type, and determining the coordinates of the first target point and the axis length of the ellipse to obtain the coordinate data corresponding to the second position.

[0058] In this possible implementation, the display effect data for the pattern type includes, but is not limited to, the ellipse fill data, such as the ellipse fill color. The coordinate data corresponding to the second position includes, but is not limited to, the coordinates of the first target point of the ellipse and its axis length. The first target point may include a point with a defined positional relationship to the ellipse or a point used to define the ellipse, such as the center point of the ellipse, the foci of the ellipse, etc. The axis length may be the major axis, minor axis, semi-major axis, and / or semi-minor axis of the ellipse, etc. When the pattern type is an ellipse, the algorithm for converting a drawn token to a pattern can be referenced as follows:

[0059] private static void drawOval(double x,double y,double r){

[0060] Graphics2D g2d=(Graphics2D)Graphics.create();

[0061] g2d.setRenderingHint(RenderingHints.KEY_ANTIALIASING,RenderingHints.VALUE_ANTIALIAS_ON);

[0062] g2d.setColor(Color.RED);

[0063] / / The coordinates of the top left corner are (x, y), and the semi-minor axis is r.

[0064] g2d.drawOval(0,0,100,100);

[0065] g2d.setColor(Color.GRAY);

[0066] / / filling

[0067] g2d.fillOval(120,100,100,150);

[0068] g2d.dispose();

[0069] }

[0070] In one possible implementation, the pattern type is a text pattern; determining the display effect data and position coordinate data of the pattern type based on the annotation method information of the target content includes: determining the font of the text pattern according to the annotation method information of the target content to obtain the display effect data of the pattern type, and determining the coordinates of the second target point of the text pattern to obtain the coordinate data corresponding to the second position.

[0071] In this possible implementation, the display effect data for the pattern type includes, but is not limited to, the font of the text. The coordinate data corresponding to the second position includes, but is not limited to, the coordinates of the second target point of the text pattern. The coordinates of the second target point can be a point or corner point on the edge of the area to be occupied by the text, for example, the lower left corner after the text is drawn. When the pattern type is a text pattern, the token-to-text conversion algorithm can be referenced as follows:

[0072] private static void drawText(String text,float x,float y){

[0073] Graphics g=BufferedImage.getGraphics();

[0074] Graphics2D g2d=(Graphics2D)g.create();

[0075] g2d.setRenderingHint(RenderingHints.KEY_ANTIALIASING,RenderingHints.VALUE_ANTIALIAS_ON);

[0076] g2d.setColor(Color.RED);

[0077] / / Font size is 50px

[0078] g2d.setFont(new Font(null,Font.PLAIN,50));

[0079] / / Coordinates refer to the bottom left corner position after the text is drawn.

[0080] g2d.drawString(text,x,y);

[0081] g2d.dispose();

[0082] }

[0083] In one possible implementation, the following steps may also be performed: in response to receiving an interpretation instruction for the material to be played, calling the trained large model to generate interpretation summary data for the material to be played, and controlling the terminal device to play the interpretation summary data; in response to receiving a question-and-answer instruction for the material to be played, calling the trained large model to generate question-and-answer results for the question-and-answer instruction, and controlling the terminal device to play the question-and-answer results.

[0084] In this possible implementation, the interpretation command for the material to be played triggers the function of calling the large model to generate interpretation summary data of the material to be played. The question-and-answer command for the material to be played triggers the function of calling the large model to generate the question-and-answer result of the question-and-answer command, thereby enabling the terminal device to communicate with users in different application scenarios. When the terminal device plays a program, it can read and understand the content being played, and it can intelligently answer questions raised by users within the scope of the program being played.

[0085] See Figure 2 The diagram shows an architecture of a terminal device that can intelligently highlight key points and answer user questions when playing programs. The following describes a possible specific implementation of this method.

[0086] The server executes any of the above-described large-model-based media playback methods. The server communicates with the terminal device. The server includes a media module, a large-model algorithm module, a highlighting algorithm module, and a program module. The terminal device has a built-in camera and microphone; its display screen is divided into a program playback area and a question-and-answer area. See also... Figure 3 The diagram illustrates a large-scale model algorithm, which includes fine-tuning and training of the AI ​​(Artificial Intelligence) large-scale model base model, implementing key highlighting functions based on the AI ​​large-scale model, implementing intelligent interpretation functions based on the AI ​​large-scale model, and implementing user interaction functions based on the AI ​​large-scale model. (See also...) Figure 4 The diagram shows a key highlighting algorithm, which includes the following steps: obtaining the location of the key content and the highlighting token, converting the highlighting token into an image, using the highlighting API, and outputting the material marked with the key content.

[0087] This method can be implemented based on the following steps:

[0088] 1) Users upload materials to be played and store them in the material module. If the uploaded material is a Word document or a PPT, the user can choose to highlight key points. For all materials stored in the material module, the "intelligent interpretation" option can be selected. 2) For materials with highlighted key points, the system automatically and asynchronously calls a large-scale algorithm to obtain the key content of the material. The obtained key content and the original material are then passed as input parameters to the highlighting algorithm. The highlighting algorithm uses the API provided by MS-Office to mark the key content of the original material. 3) The program module allows users to set layout templates, materials to be played, and playback times in the program editor, and then send instructions to the terminal device via the MQTT protocol. 4) After receiving the instructions, the terminal device calls the server's interface to obtain detailed program information and the materials to be played, and then plays the program. For programs without intelligent interpretation, there is no question-and-answer area, and the program playback area fills the entire screen.

[0089] The following explanation is provided regarding the execution of steps 1)-4) above:

[0090] The materials module supports uploading common materials (images, audio, adapters, documents). For uploaded Word documents or PPT presentations, users can select and highlight key points. For all uploaded materials, a smart interpretation option can be selected. For materials with highlighted points enabled, a large-scale model algorithm will be asynchronously invoked to analyze the key content of the entire document, outputting the location and pattern of the key content. The large-scale model algorithm includes: fine-tuning training of the AI ​​large-scale model base model; implementing the highlighted point function based on the AI ​​large-scale model; implementing the smart interpretation function based on the AI ​​large-scale model; and implementing user interaction functions based on the AI ​​large-scale model. The highlighted point algorithm uses the output of the large-scale model's analysis of key content as input parameters, utilizing open interfaces provided by MS-Office to mark the corresponding content in the raw materials, i.e., highlighting key content by calling the highlighted point algorithm API. The program module mainly involves program production, including a program editor. Users can layout the playback area, design which part plays which material, and then set the playback time. Processed materials are turned into programs: users can set the layout, drag and drop materials to be played into the editor, and then save the program.

[0091] The prepared program is distributed to selected terminal devices via the MQTT protocol, and playback instructions are sent to these devices. Upon receiving the playback instructions, the terminal devices retrieve detailed program information from the server-side interface and play the program according to the set schedule. For materials without intelligent interpretation enabled, the program plays full-screen. For materials with intelligent interpretation enabled, the program also plays full-screen when no user question is detected; when a user question is detected, the program is automatically split into a program playback area and a question-and-answer area when responding to the question. During program playback, the terminal device's built-in pedestrian detection system initiates a voice interaction when someone is detected watching. It then calls the speech-to-text API to convert the speech to text, and then calls the server-side interface as input to invoke a large-scale algorithm, returning the answers within the scope of the played material to the terminal device for display.

[0092] The above method can intelligently highlight key points in user-uploaded materials, such as Word documents or PPT presentations, using large-scale model algorithms. When playing programs on terminal devices, it adds audience detection, which can detect whether there are viewers watching. When viewers are watching, it can proactively initiate communication with users and answer questions raised by viewers within the scope of the content being played.

[0093] This application also provides a method for training large models, such as... Figure 5 The diagram shown is a flowchart of a large model training method according to an embodiment of this application. The method may include:

[0094] In step S501, first playback material training data including pattern tag data is obtained; wherein, the pattern tag data is used to identify target content in the first playback material training data.

[0095] In this embodiment, the first playback material training data includes various playback materials, and the data should be as rich as possible. Pattern label data is obtained by marking the playback materials. For example, various important highlighting patterns are defined, such as red text indicating importance, underlines indicating text reminders, ellipses indicating pattern reminders, etc. Professionals understand the materials and highlight them according to the defined patterns to obtain the labeling results, which are the pattern label data. A unique token is designed for each highlighting pattern, and each highlighting pattern in the training data is represented as <unique token> + position and size.

[0096] In step S502, a preset large model is trained using the first playback material training data to obtain a trained large model, and the material playback method based on the large model described above is implemented based on the trained large model.

[0097] In this embodiment, SFT (Supervised Fine-Tuning) training is performed on the training data based on the already fine-tuned base model, enabling it to predict the location of key content and its circled token.

[0098] After training is completed, the model is deployed. For new test material, it is input into the trained SFT model. After the model predicts the location of the key content and the circle token, the circle token is converted into the circle pattern it represents. The circle API interface is called to circle the key content location of the material accordingly.

[0099] This application provides a large model training method. According to this application, the target content included in the material to be played identified by the large model can be labeled to obtain labeled data labeled according to pattern type at the first position. Finally, the terminal device is controlled to play the material to be played including the labeled data so that the user can obtain the important content in the material played by the terminal device in a timely manner.

[0100] Considering that existing large-scale AI models often lack sufficient understanding of playback-type materials, fine-tuning training is needed to enhance this ability. In one possible implementation, fine-tuning the basic model of the large-scale AI model can be performed according to the following steps: training a preset large-scale model using the first playback material training data, including: using an image encoder to convert the image data in the first playback material data into text data to obtain adjusted first playback material training data; and training the preset large-scale model using the adjusted first playback material training data.

[0101] In this possible implementation, training data preparation involves preparing a large amount of playback material, cleaning and filtering it to form a playback-class training set, and then adding this training set to the training set of a regular multimodal large model to obtain the total training set. The large model can adopt the same transformer structure as the GPT model. To enable the model to understand images, an image encoder is also added to map image tokens to text embedding space.

[0102] It should be noted that fine-tuning the image encoder mainly involves optimizing the contrastive learning loss function between the image encoder and the text encoder. The objective function is as follows:

[0103]

[0104] Where N represents the number of training samples, x i Let y represent the image encoder features of the i-th sample. i Let x represent the text encoder features of the i-th sample. j Let y represent the text features of the j-th sample. j This represents the text features of the j-th sample.

[0105] This function can be used to align images and text, enabling large language models to understand image features.

[0106] Fine-tuning the large language model, where the model predicts the next token, uses cross-entropy loss as the objective function.

[0107]

[0108] Where T represents the total number of tokens, y t Let y represent the actual value of the t-th token. <t Let represent the predicted values ​​of the first t tokens, and x represent the image features. This can provide a supervisory signal for large model predictions (the more consistent the predicted token probability distribution is with the true probability distribution, the smaller the cross-entropy loss), making the predictions more accurate.

[0109] In one possible implementation, the following operations can also be performed: acquiring second playback material training data including summary tag data and third playback material training data including question-answer result tag data; and training the preset large model using the second playback material training data and the third playback material training data respectively.

[0110] In this embodiment of the application, the intelligent interpretation function based on the AI ​​large model can be implemented according to the following steps:

[0111] 1. Prepare a training set consisting of various playback materials, making it as comprehensive as possible;

[0112] 2. Pseudo-annotation generation: Based on the AI ​​basic model, a summary of the playback material on each page is generated as pseudo-annotations;

[0113] 3. Manual verification: Professionals correct the pseudo-annotations on the playback materials to obtain a high-quality interpretation summary, which serves as the annotation result for the training data;

[0114] 4. Model Training: Load the basic AI model as the initial model, and fine-tune the model using the playback materials and their annotations so that it can accurately predict the interpretation summary of each page of playback materials.

[0115] After training is completed, the model is deployed: For the playback materials uploaded by the customer, the newly trained AI model is called to perform a summary interpretation of each page of the material, generate a text script, and call the text-to-audio conversion tool to generate the corresponding interpretation audio.

[0116] To implement user interaction functions based on a large AI model, the following steps can be followed:

[0117] To make the current AI model's question-and-answer function more closely resemble the playback scenario, it needs to be fine-tuned during training.

[0118] 1. Training data collection: Collect a large amount of playback material and clean, filter and organize it.

[0119] 2. Manual annotation: For each training material, we design questions that the audience might ask, and ask professionals to provide appropriate answers to obtain annotated data.

[0120] 3. Model Training: Based on the AI ​​base model, train the above training materials and their annotations to make the model's answers to each question closer to the answers in the annotations.

[0121] Model Deployment: Deploy the trained model into the system. Usage Flow:

[0122] (1) After playing the material, if any audience member asks a question, the question will be converted into text through speech recognition.

[0123] (2) Input the question text into the fine-tuned AI model, and the model outputs the answer to the question.

[0124] The answers are transcribed into audio and played back as a Q&A session with the audience.

[0125] Based on the embodiments of this application, programs are played according to settings when no one is watching, and can proactively initiate communication and answer user questions within the scope of the played content when someone is watching. This mainly includes: users uploading playback content to the material module; for PPT or World presentations, users can choose to enable highlighting; for any playback content, intelligent interpretation can be enabled. For materials with highlighting enabled, the server asynchronously calls the large model algorithm to obtain the key objects of the material, and uses the highlighted key objects and the original material as input parameters to the highlighting module to mark the key content of the original material. When the large model detects an audience within a 3-meter radius directly in front of the playing program, it can proactively initiate communication with the user. For user questions, the audio is captured by the device's microphone, converted into text on the terminal device, and transmitted to the server as input parameters for the large model algorithm. The answer is then sent to the device's question-and-answer area for display. This effectively solves the problem that traditional terminal devices cannot communicate with users or resolve user questions while playing programs; simultaneously, it can intelligently highlight key content in the played materials.

[0126] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the methods described in the above embodiments. The number of memories 601 and processors 602 can be one or more.

[0127] The electronic device also includes:

[0128] The communication interface 603 is used to communicate with external devices and perform data exchange and transmission.

[0129] If the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0130] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0131] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0132] This application provides a computer program product, wherein the computer program product includes a computer program, which, when executed by a processor, implements the method provided in this application embodiment.

[0133] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0134] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0135] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0136] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0137] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0138] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0139] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0140] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0141] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0142] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0143] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0145] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / text> < / line> < / oval>

Claims

1. A method for playing back media based on a large model, comprising: The system acquires the material to be played, and in response to receiving the annotation instruction for the material to be played, it calls the trained large model to identify the target content in the material to be played, and obtains the first position and annotation method information of the target content; The pattern type used to identify the first position is determined based on the annotation method information of the target content; Generate annotation data corresponding to the target content in the material to be played according to the pattern type and the first position; The control terminal device plays the material to be played, which includes the labeled data.

2. The method according to claim 1, wherein, Determining the pattern type used to identify the first location based on the annotation method information of the target content includes: Based on the annotation method information of the target content, determine the display effect data and second position of the pattern type; The pattern type for identifying the first position of the target content is determined based on the display effect data and the second position.

3. The method according to claim 2, wherein, The pattern type is a line segment; Based on the annotation method information of the target content, the display effect data and second position of the pattern type are determined, including: The stroke width of the line segment is determined based on the annotation method information of the target content, and the display effect data of the pattern type is obtained. The endpoints of the line segment are determined, and the coordinate data corresponding to the second position is obtained.

4. The method according to claim 2, wherein, The pattern type is an ellipse; the display effect data and second position of the pattern type are determined based on the annotation method information of the target content, including: Based on the annotation information of the target content, the filling data of the ellipse is determined to obtain the display effect data of the pattern type, and the coordinates and axis length of the first target point of the ellipse are determined to obtain the coordinate data corresponding to the second position.

5. The method according to claim 2, wherein, The pattern type is a text pattern; Based on the annotation method information of the target content, the display effect data and position coordinate data of the pattern type are determined, including: The font size of the text pattern is determined based on the annotation method information of the target content, and the display effect data of the pattern type is obtained. The coordinates of the second target point of the text pattern are determined, and the coordinate data corresponding to the second position is obtained.

6. The method according to claim 1, wherein, Also includes: In response to receiving a decoding instruction for the material to be played, the trained large model is invoked to generate decoding summary data for the material to be played, and the terminal device is controlled to play the decoding summary data. In response to receiving a question-and-answer instruction for the material to be played, the trained large model is invoked to generate a question-and-answer result for the instruction, and the terminal device is controlled to play the question-and-answer result.

7. A method for training large models, comprising: Acquire first playback material training data including pattern tag data; wherein, the pattern tag data is used to identify target content in the first playback material training data; A preset large model is trained using the first playback material training data to obtain a trained large model, and the method described in any one of claims 1-6 is implemented based on the trained large model.

8. The method according to claim 7, wherein, Training a pre-set large model using the first playback material training data includes: Using an image encoder, the image data in the first playback material data is converted into text data to obtain the adjusted first playback material training data; The pre-set large model is trained using the adjusted first playback material training data.

9. The method according to claim 7, wherein, Also includes: Acquire training data for the second playback material, including summary tag data, and training data for the third playback material, including question and answer result tag data; The preset large model is trained using the training data from the second playback material and the training data from the third playback material, respectively.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-9.

11. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

12. A computer program product, wherein, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-9.

Images