Explanation of Problems: Methods, Systems, Equipment, and Program Products

Abstract

This application provides a method, system, device, and program product for explaining problem-solving questions. The method includes: determining a compressed resolution that matches the resolution of the problem image, and encoding the problem image after compression according to the compressed resolution to obtain initial image features; performing cross-attention operations on the initial image features and the initialized problem features to obtain effective problem features; wherein, the initialized problem features are the initial values ​​of the effective problem features; the effective problem features include effective problem information in the initial image features; generating the solution content of the problem based on the effective problem features; outputting semantic blocks one by one based on the effective problem features and the solution content of the problem; wherein, each semantic block is used to represent a step in the explanation of the problem; and generating an explanation video based on each output semantic block. This application can improve the speed of problem explanation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, system, device, and program product for explaining a topic. Background Technology

[0002] The application of question-and-answer assistance for learning is becoming increasingly widespread. When users encounter various types of questions during their daily studies, they often struggle to quickly and accurately arrive at the answer and understand the knowledge points involved due to the complexity of the question's wording, the abundance of formulas, or their own misunderstandings. Therefore, many users take clear pictures of the questions and use smart applications or online learning platforms to obtain explanations of the questions presented in the images.

[0003] However, after users photograph and upload the problem, they often have to wait a considerable amount of time to receive the explanation. Sometimes, even simple problems may take several minutes or longer to get a result, while for more complex problems requiring in-depth analysis and calculation, the waiting time can be even longer. Currently, the explanation speed is relatively slow. Summary of the Invention

[0004] This application provides a method, system, device, and program product for explaining problems, in order to improve the speed of problem explanation.

[0005] According to a first aspect of the embodiments of this application, a method for explaining problems is provided, including:

[0006] Determine a compressed resolution that matches the resolution of the question image, and encode the compressed question image according to the compressed resolution to obtain initial image features;

[0007] A cross-attention operation is performed on the initial image features and the initial question features to obtain effective question features; wherein, the initial question features are the initial values ​​of the effective question features; the effective question features include the effective question information in the initial image features;

[0008] Based on the aforementioned effective question characteristics, the solution content for the question is generated;

[0009] Based on the valid question features and the answer content of the question, semantic blocks are output one by one; wherein each semantic block is used to represent an explanation step of the question;

[0010] For each semantic block output, an explanatory video is generated based on the output semantic block.

[0011] According to a second aspect of the embodiments of this application, a problem-solving system is provided, comprising:

[0012] The question-answering module is used to determine a compressed resolution that matches the resolution of the question image, and to encode the compressed question image according to the compressed resolution to obtain initial image features; to perform cross-attention operation on the initial image features and the initialized question features to obtain effective question features; wherein, the initialized question features are the initial values ​​of the effective question features; the effective question features include the effective question information in the initial image features; and to generate the answer content of the question based on the effective question features.

[0013] An interactive video generation module is used to output semantic blocks one by one based on the valid question features and the answer content of the question; wherein each semantic block is used to represent an explanation step of the question; and an explanation video is generated based on the output semantic block when each semantic block is output.

[0014] According to a third aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor;

[0015] The memory is connected to the processor and is used to store programs;

[0016] The processor is used to implement the problem explanation method as described in the first aspect by running the program in the memory.

[0017] According to a fourth aspect of the embodiments of this application, a computer program product is provided, including computer program instructions, which, when executed by a processor, cause the processor to perform the problem explanation method as described in the first aspect.

[0018] In this application, a compressed resolution matching the resolution of the question image is determined, and the question image is compressed and encoded according to the compressed resolution to obtain initial image features. The compressed resolution can be dynamically adjusted according to the resolution of the question image. Compared to the prior art which encodes the question image at a fixed resolution, this reduces invalid encoding and thus reduces the amount of data in the initial image features. Cross-attention operations are performed on the initial image features and the initialized question features to obtain effective question features. The initialized question features are the initial values ​​of the effective question features. Effective question features include the effective question information from the initial image features, allowing for the extraction of effective question information and eliminating interference from background information. Furthermore, the amount of data in the effective question features is less than that in the initial image features. Based on the effective question features, the answer to the question is generated. In the process of generating the answer, only the effective question features are used, reducing the amount of data in the image features involved in the answer, increasing the speed of generating the answer, and further improving the speed of question explanation.

[0019] The amount of data for effective question features is less than the amount of data for initial image features. Based on the effective question features and the answer content, semantic blocks are output one by one. Only effective question features are used in the generation of semantic blocks, which reduces the amount of data from image features involved in generating semantic blocks and improves the speed of semantic block generation. Furthermore, this application outputs semantic blocks one by one based on effective question features and the answer content. Each semantic block represents one step in the explanation of the question. When a semantic block is output, an explanation video is generated based on the output semantic block. This eliminates the need to wait for all explanation steps of the question to be generated before generating the explanation video. It allows the entire explanation process of the question to be broken down and output step by step, generating explanation videos, reducing user waiting time and improving the speed of question explanation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a problem-solving method provided in an embodiment of this application.

[0022] Figure 2 This is a flowchart illustrating step 101 provided in an embodiment of this application.

[0023] Figure 3 This is a schematic diagram illustrating the method of explaining problems using traditional image encoding in existing technologies.

[0024] Figure 4 This is a schematic diagram of a problem-solving method provided in an embodiment of this application.

[0025] Figure 5 This is a schematic diagram of a process for generating explanatory videos based on output semantic blocks, as provided in an embodiment of this application.

[0026] Figure 6 This is a schematic diagram illustrating the reinforcement learning process of an evaluation model and an interactive video model provided in the embodiments of this application.

[0027] Figure 7 This is a schematic diagram illustrating a process for generating an explanatory video, as provided in an embodiment of this application.

[0028] Figure 8 This is a schematic diagram of a problem-solving method provided in an embodiment of this application.

[0029] Figure 9This is a schematic diagram of the structure of a question explanation system provided in an embodiment of this application.

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The application of question-and-answer assistance for learning is becoming increasingly widespread. When users encounter various types of questions during their daily studies, they often struggle to quickly and accurately arrive at the answer and understand the knowledge points involved due to the complexity of the question's wording, the abundance of formulas, or their own misunderstandings. Therefore, many users take clear pictures of the questions and use smart applications or online learning platforms to obtain explanations of the questions presented in the images.

[0032] Currently, the main technical solutions for answering questions to aid learning include the following:

[0033] (a) Search Scheme

[0034] The search solution involves searching the question bank for images taken by the user, outputting the analysis and answers of the retrieved questions, and playing video explanations from teachers for a few search results.

[0035] The search solution relies on a question bank. Retrieving questions and answers from the question bank requires maintaining a large question bank system. When the question bank's coverage is insufficient or it is not updated in a timely manner, the user's original question cannot be found. In this case, similar questions are usually output, hoping that the user can summarize the knowledge and experience from them and apply it to solving their own questions. This is equivalent to throwing the problem back to the user.

[0036] The current search solutions are resource-intensive and lack interactivity. Some solutions use pre-recorded teacher explanation videos to improve tutoring quality, but this requires a huge investment of manpower and a large amount of video resources. From the perspective of actual user experience, not only is the coverage of questions in the explanation videos low, but the entire process also lacks interaction with students. Teachers cannot understand the students' understanding, and students cannot communicate their questions to the teachers.

[0037] (II) Generation Scheme

[0038] The solution generation process often involves solving the problem using a large model, and then outputting the solution steps in text form.

[0039] The generated solutions have a high error rate. The large model uses an autoregressive decoding method to solve the problem. Because the large model itself is a probabilistic model, the entire process is random and uncontrollable, leading to errors, fabrications, and contradictions in the output. Furthermore, as the errors in the decoding process accumulate, the final output will deviate significantly from the answer.

[0040] The explanation of the generated solutions is slow. Multimodal large models consist of image encoding and text autoregressive decoding. The response speed of multimodal large models is affected by image resolution, image encoder efficiency, language large model decoding efficiency, and decoded text length. Although there are techniques for compressing input based on image feature similarity, this can lead to the loss of similar text and chart content, resulting in misunderstandings of the questions. The similarity threshold setting and fusion method are also highly dependent on engineering experience, and often a single setting is only applicable to one scenario, exhibiting weak generalization. With the development of imaging equipment and the application of deep thinking modes in large models, image resolution and decoding length have increased exponentially, leading to increasingly slower response times, sometimes exceeding 10 minutes for a single question.

[0041] The generated solutions are limited in modality. While the large model can provide relatively detailed solutions to user questions, the solution steps are presented entirely in text form. There is no audio explanation to alleviate the reading burden, and when it comes to difficult and complex questions, there is no gradual presentation of necessary hints, drafts, and annotations, resulting in a poor learning experience for users.

[0042] Currently, with existing quiz-based learning solutions, users often have to wait a considerable amount of time after photographing and uploading a question to receive an explanation. Sometimes, even simple questions can take several minutes or longer to get a result, while more complex questions requiring in-depth analysis and calculation can take even longer. The current explanation speed is also quite slow.

[0043] Therefore, in order to improve the speed of problem explanation, this application provides a problem explanation method, system, device and program product.

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] Exemplary Implementation Environment

[0046] The title explanation method according to the embodiments of this application can be executed by electronic devices such as terminal devices or servers. Terminal devices can be user devices, mobile devices, computing devices, wearable devices, etc., and servers can be independent physical servers, server clusters composed of multiple physical servers, or cloud servers capable of cloud computing. This method can be implemented by a processor calling computer-readable program instructions stored in memory.

[0047] Exemplary methods

[0048] Please see Figure 1 In one exemplary embodiment, a method for explaining a problem is provided. For example... Figure 1 As shown, the process of explaining the problem mainly includes:

[0049] Step 101: Determine the compressed resolution that matches the resolution of the question image, and encode the compressed question image according to the compressed resolution to obtain the initial image features.

[0050] In the exemplary embodiment, the problem image refers to an image obtained by the user taking a picture of the problem. Users can input problem images by taking pictures, which is more convenient than text input and avoids situations where complex formulas cannot be input.

[0051] In the exemplary embodiment, the compressed resolution refers to the resolution of the image obtained after compressing the question image. Then, the image obtained after compressing the question image is encoded to obtain the initial image features.

[0052] In some embodiments, such as Figure 2 As shown, step 101 includes:

[0053] Step 201: Determine the downsampling factor based on the resolution of the title image and the resolution of the preset area.

[0054] The preset area consists of multiple grids.

[0055] In the exemplary embodiment, the resolution of different question images may be the same or different, depending on the user's shooting situation and the question itself.

[0056] In the exemplary embodiment, the resolution of the preset area is a fixed resolution. For example, the resolution of each grid is 10 pixels × 10 pixels. The preset area has 100 grids in the length direction and 100 grids in the width direction. The preset area is composed of 10,000 grids, so the resolution of the preset area is 1,000 pixels × 1,000 pixels.

[0057] In an exemplary embodiment, step 201 may include: determining a first ratio obtained by dividing the length of the question image in pixels by the length of the preset region in pixels; determining a second ratio obtained by dividing the width of the question image in pixels by the width of the preset region in pixels; rounding the target ratio up to obtain the downsampling factor; wherein the target ratio is the maximum value of the first ratio and the second ratio. This enables downsampling of the question image according to the downsampling factor, resulting in a downsampled image with a resolution lower than the resolution of the preset region, and the downsampled image can fill the preset region as much as possible, avoiding the loss of important information in the downsampled image due to an excessively large downsampling factor.

[0058] For example, the resolution of the question image is 5000 pixels × 4000 pixels, the resolution of the preset area is 1000 pixels × 1000 pixels, and the downsampling factor is 5; the resolution of the question image is 4000 pixels × 3000 pixels, the resolution of the preset area is 1000 pixels × 1000 pixels, and the downsampling factor is 4.

[0059] Step 202: Based on the downsampling factor, downsample the image to obtain the downsampled image.

[0060] Step 203: Select a target region within the preset region that matches the resolution of the downsampled image.

[0061] The target area consists of at least one grid.

[0062] For example, if the image resolution is 5000 pixels × 4000 pixels, and the downsampling factor is 5, the downsampled image resolution is 1000 pixels × 800 pixels. Each grid cell has a resolution of 10 pixels × 10 pixels. The downsampled image has 100 grid cells in the length direction and 80 grid cells in the width direction. Since the number of grid cells in both the length and width directions of the downsampled image is an integer, it perfectly fills the entire grid area. In this case, the target region's resolution matches the downsampled image's resolution. Therefore, the target region matching the downsampled image's resolution has 100 grid cells in the length direction and 80 grid cells in the width direction. In other words, when the number of grid cells in both the length and width directions of the downsampled image is an integer, the resolution of the downsampled image is determined as the resolution of the target region.

[0063] For example, if the resolution of the image in the question is 4200 pixels × 3240 pixels, and the downsampling factor is 5, the resolution of the downsampled image is 840 pixels × 648 pixels. Each grid cell has a resolution of 10 pixels × 10 pixels. The downsampled image has 84 grid cells in the length direction and 64.8 grid cells in the width direction. At least one of the number of grid cells in the length and width directions of the downsampled image is a decimal, and it does not fill the entire grid area. In this case, the resolution of the target area is inconsistent with the resolution of the downsampled image. Therefore, the target area that matches the resolution of the downsampled image has 84 grid cells in the length direction and 65 grid cells in the width direction. Specifically, if at least one of the grid counts in the length and width directions of the downsampled image is a decimal, the grid count in the first direction of the downsampled image is rounded up to obtain the grid count in the first direction of the target region; the grid count in the second direction of the downsampled image is determined as the grid count in the second direction of the target region; the resolution of the target region is determined based on the grid count in the first and second directions of the target region; wherein, the first direction is the direction in which the grid count in the downsampled image is a decimal, and the first direction includes at least one of the length and width directions; the second direction is the direction in which the grid count in the downsampled image is an integer.

[0064] Step 204: Determine the resolution of the target area as the compressed resolution.

[0065] Step 205: Adjust the resolution of the downsampled image to the compressed resolution to obtain the adjusted image, and encode the adjusted image to obtain the initial image features.

[0066] Based on the resolution of the question image and the resolution of the preset region, a downsampling factor is determined. The downsampling factor changes with the resolution of the question image. The question image is downsampled according to the downsampling factor to obtain a downsampled image. The resolution of the downsampled image also changes with the resolution of the question image. Within the preset region, a target region matching the resolution of the downsampled image is selected, and the resolution of the target region is determined as the compressed resolution. The resolution of the downsampled image is adjusted to the compressed resolution to obtain an adjusted image. The adjusted image is then encoded to obtain the initial image features. A dynamic resolution and adaptive sampling encoding method is adopted, which can ensure that the adjusted image is not distorted. Compared with encoding the question image at a fixed resolution, it can also reduce invalid encoding and reduce the amount of data for the initial image features.

[0067] In some other embodiments, step 101 includes: dividing the resolution of the question image by a preset downsampling factor to obtain the compressed resolution, and encoding the compressed question image according to the compressed resolution to obtain initial image features.

[0068] Step 102: Perform cross-attention operation on the initial image features and the initial question features to obtain effective question features.

[0069] The initial question features are the initial values ​​of the valid question features; the valid question features include the valid question information in the initial image features.

[0070] In an exemplary embodiment, step 102 may include: inputting initial image features and initialized question features into the Adaptor network to obtain effective question features. The Adaptor network is an adaptation network employing a transformer architecture. Initialized question features are initialized query features, which are meaningless image features. Effective question features are effective query features, which are image features in the initial image features that include valid question information such as text and charts. Cross-attention operation refers to the cross-attention operation performed on the initial image features and initialized question features to obtain effective question features. This operation is used in both the inference and training processes of the Adaptor network to extract effective question features from the initial image features.

[0071] Step 103: Based on the characteristics of valid questions, generate the answer content for the questions.

[0072] In an exemplary embodiment, the solution to a question includes the explanation of the question and the answer.

[0073] In an exemplary embodiment, step 103 may include: inputting valid question features into a large language model to obtain the answer to the question.

[0074] In an exemplary embodiment, such as Figure 3 The diagram shown illustrates a problem-solving method using traditional image encoding in existing technologies, which is also the traditional method for processing large multimodal models. Figure 3In this approach, the question image is input into an image encoder, which encodes it at a fixed resolution to obtain image features. Traditional fixed-resolution encoding in multimodal large-scale models resizes the question image to a fixed size before encoding. To avoid losing text and chart information, a relatively large fixed size is typically used. Furthermore, to adapt to different aspect ratios of question images, this fixed size is often square, such as 2048 pixels × 2048 pixels, which introduces a large amount of invalid encoding. This approach compresses image features based on similarity, resulting in compressed image features. These compressed features are then input into a large language model to obtain the question's parsing and answer. Since only a small amount of text and charts in the question image are useful information, while a large amount of background is irrelevant, this characteristic can be used to compress the output features of the image encoder. Existing techniques fuse highly similar features in the image, but this method ignores a large amount of similar text and charts in the question image, leading to the loss of question information.

[0075] In an exemplary embodiment, such as Figure 4 The diagram illustrates a question explanation method provided in this embodiment. The question image is input into an image encoder, which encodes it according to steps 201 to 205 to obtain initial image features. A dynamic resolution and adaptive sampling encoding method is used, ensuring that the adjusted image is not distorted. Compared to encoding the question image at a fixed resolution, this method also reduces invalid encoding and lowers the data volume of the initial image features. The initial image features and initial question features are input into an adaptation network to obtain effective question features. These effective question features are then input into a large language model to obtain the question's parsing and answer. The adaptation network and effective question features are iteratively optimized with the large language model, ultimately extracting effective question features that closely reflect the question's valid information and eliminate background interference. Because the data volume of effective question features is much smaller than that of initial image features, only the effective question features are input into the large language model, reducing the decoding length of the large language model, improving response speed, and increasing the speed of question explanation. The image encoding uses dynamic resolution and compression features, and the text decoding only involves a small amount of parsing and answering, resulting in high efficiency and an overall time controllable within 1 second.

[0076] In the exemplary embodiment, steps 101 to 103 can be implemented using a question-answering model. The question image is input into the model to obtain the answer. This model is trained on a massive question bank, enabling it to form a strong memory of the questions and answers, thus achieving the same effect as the retrieval scheme. Furthermore, the model is not rote memorization; as a large language model, it possesses strong generalization capabilities, allowing it to provide correct results even when the question type or details differ from the question bank. The specialized training ensures reliable answers and explanations, reducing the difficulty and error probability of subsequent explanation steps. It is possible to answer all user questions without relying on a question bank, as the model is trained on massive amounts of question-answering data, guaranteeing accuracy consistent with the question bank.

[0077] In an exemplary embodiment, the question-answering model includes Figure 4 Image encoders, adaptation networks, and large language models are used in this study.

[0078] In this application, a compressed resolution matching the resolution of the question image is determined, and the question image is compressed and encoded according to the compressed resolution to obtain initial image features. The compressed resolution can be dynamically adjusted according to the resolution of the question image. Compared to the prior art which encodes the question image at a fixed resolution, this reduces invalid encoding and thus reduces the amount of data in the initial image features. Cross-attention operations are performed on the initial image features and the initialized question features to obtain effective question features. The initialized question features are the initial values ​​of the effective question features. Effective question features include the effective question information from the initial image features, allowing for the extraction of effective question information and eliminating interference from background information. Furthermore, the amount of data in the effective question features is less than that in the initial image features. Based on the effective question features, the answer to the question is generated. In the process of generating the answer, only the effective question features are used, reducing the amount of data in the image features involved in the answer, increasing the speed of generating the answer, and further improving the speed of question explanation.

[0079] Step 104: Based on the valid question features and the answer content of the question, output semantic blocks one by one.

[0080] Each semantic block represents a step in explaining the question.

[0081] Step 105: For each semantic block output, generate an explanatory video based on the output semantic block.

[0082] In this application, the amount of data for effective question features is less than the amount of data for initial image features. Semantic blocks are output sequentially based on the effective question features and the answer content. Only effective question features are used in the semantic block generation process, reducing the amount of data from image features involved in generating semantic blocks and improving the speed of semantic block generation. Furthermore, this application outputs semantic blocks sequentially based on effective question features and the answer content. Each semantic block represents one step in the explanation of the question. When a semantic block is output, an explanation video is generated based on the output semantic block. This eliminates the need to wait for all explanation steps of the question to be generated before generating the explanation video. It allows the entire explanation process of the question to be broken down and output step-by-step, reducing user waiting time and improving the speed of question explanation.

[0083] In the exemplary embodiment, each semantic block is distinguished by a marker. When the model decodes the first marker, it indicates that a complete semantic block has been completed. At this time, image rendering and speech synthesis are started to avoid the user waiting.

[0084] Given the rich content of the complete explanation steps for the question and the long text decoding time of the large language model, generating the complete explanation steps based on the effective question features and the solution content before outputting them would inevitably result in a user wait time of several minutes. To avoid the lengthy process of generating the explanation video and causing user wait time, the complete explanation process for the question will be broken down into explanation steps and output. Each explanation step is defined as a semantic block. When each semantic block is output, an explanation video is generated based on the output semantic block, reducing user wait time.

[0085] In the exemplary embodiment, steps 101 to 103 can be implemented using a question-answering model, by inputting the question image into the model to obtain the answer. Steps 101, 102, and 104 can be implemented using an interactive video model, by inputting the question image and the answer to the model to obtain each semantic block. Step 105 is implemented outside the interactive video model by calling a speech synthesis tool and an image rendering tool on the semantic blocks.

[0086] In an exemplary embodiment, the question-answering model includes Figure 4 The image encoder, adaptation network, and large language model in the interactive video model include... Figure 4 The interactive video model includes an image encoder, an adaptation network, and a large language model. The large language model in the interactive video model takes valid question features and the answer content as input, and outputs semantic blocks. The image encoding method is consistent with that of the question-answering model, which can significantly reduce the response time and improve the response speed of the interactive video model.

[0087] In some embodiments, semantic blocks include explanatory content, punctuation content, and blackboard writing content.

[0088] In the exemplary embodiment, the explanatory content refers to the explanatory text. The marked content includes type and coordinates; the type is highlight, underline, circle, etc., and the coordinates correspond to the top-left and bottom-right corners of a rectangle on the title image. The blackboard content refers to the text written on the blackboard.

[0089] In some embodiments, such as Figure 5 As shown, step 105 generates an explanatory video based on the output semantic blocks, including:

[0090] Step 501: Perform speech synthesis on the explanatory content in the output semantic block to generate explanatory audio.

[0091] Step 502: Render the image based on the plotted content in the output semantic block to obtain the plotted result.

[0092] The plotting results are used to plot the images in the question.

[0093] Step 503: Render the image based on the whiteboard content in the output semantic block to obtain the whiteboard image.

[0094] Step 504: Add the plotted results and blackboard image to the question image to generate the explanation image.

[0095] Step 505: Generate an explanatory video based on the explanatory audio and explanatory images.

[0096] The marked content and blackboard writing in the semantic block are rendered into the question image to generate an explanation image. The explanation content in the semantic block is synthesized and broadcast. The output includes not only the audio explanation, but also the marked prompts on the question and the blackboard writing steps, simulating the teaching process of a teacher explaining and writing at the same time. This achieves multimodal explanation, which can attract users' attention more and improve the learning effect.

[0097] In some embodiments, the semantic block further includes rhetorical questions. Step 505 includes: performing speech synthesis based on the rhetorical questions in the output semantic block to generate rhetorical question audio; and generating an explanatory video based on the explanatory audio, explanatory image, and rhetorical question audio.

[0098] During the explanation, questions or counter-questions will be posed to the user regarding the knowledge points being discussed. This not only prevents the user from getting distracted but also tests the user's comprehension. The explanation process is highly interactive with the user.

[0099] In the exemplary embodiment, Table 1 shows the definition of semantic blocks.

[0100] Table 1

[0101]

[0102] In Table 1, Type and Content are required; other content can be left blank. The interactive video model strictly follows the explanation logic to output semantic blocks. That is, the first semantic block must be Type = reading the question, followed by several semantic blocks of Type = explaining the question, and then the semantic block of Type = answer and summary. Considering that reading the question is relatively easy and almost error-free, and in order to respond to users as quickly as possible, the Mark, Display, and Question of the first semantic block are all defaulted to being incomplete, and the evaluation model is not called.

[0103] In some embodiments, the question explanation method further includes: obtaining the user's response text corresponding to the rhetorical question; and generating the correctness judgment result and explanation content of the user's response text based on the rhetorical question and the user's response text corresponding to the rhetorical question.

[0104] In an exemplary embodiment, the rhetorical question and the corresponding user response text are input into the question-and-answer model to obtain the correctness judgment result and explanation content of the user response text.

[0105] During the explanation, questions or counter-questions will be posed to the user regarding the relevant knowledge points. For the user's response text corresponding to the counter-question, the system will output the correctness judgment result and explanation of the response text, which can help the user understand the comprehension and better comprehend the corresponding knowledge points. The explanation process is highly interactive with the user.

[0106] In some embodiments, the question explanation method further includes: when it is determined that a user interrupts to ask a question, obtaining the content of the user interruption question, pausing the operation of speech synthesis and image rendering on the output semantic block; and generating response content corresponding to the content of the user interruption question.

[0107] In an exemplary embodiment, the user interrupts the question and inputs the question into the question-answering model to obtain the response content corresponding to the user interrupts the question.

[0108] In an exemplary embodiment, obtaining the content of a user's interrupted question may include: performing speech recognition on the audio of the user's interrupted question to obtain the content of the user's interrupted question.

[0109] When a user has questions about a semantic block, they can interrupt the current explanation and ask a question at any time. When a user interrupts to ask a question, the speech synthesis and image rendering of the output semantic block are paused. At this time, the question-answering model intervenes to provide a response, and then the speech synthesis and image rendering of the output semantic block are restarted. The explanation process is highly interactive with the user.

[0110] In the exemplary embodiment, the question-answering model has two parts of input and output. One part of the input consists of the question and the corresponding user response text, and the output is the correctness judgment and explanation of the user response text. The other part of the input consists of the user interrupting the question, and the output is the response to the interrupted question. Therefore, the question-answering model is trained from two types of data: one is the feedback data, which consists of the input question and answer, and the output is the correctness judgment and explanation of the answer. This function is used to judge the question and response in interactive video. The other part is the question-answering data, which consists of the question and the response to the question. This function is used to judge the interrupted question.

[0111] In the exemplary embodiment, the question-answering model supports multi-turn interactions. If the user still has questions about the output, they can ask follow-up questions in multiple rounds. The question-answering model will provide appropriate responses based on historical dialogues and questions, such as reducing the difficulty or refining the process. In addition, in case the user's problem is not resolved in the end, a feedback button is designed. After the user clicks it, their question will be included as an optimization target for the next version of the question-answering model.

[0112] In some embodiments, step 105, generating an explanatory video based on the output semantic block, includes: inputting the output semantic block into an evaluation model to obtain an evaluation result of the output semantic block; generating an explanatory video based on the output semantic block if the evaluation result indicates that the output semantic block is correct; and correcting the output semantic block to obtain a corrected semantic block and generating an explanatory video based on the corrected semantic block if the evaluation result indicates that the output semantic block is incorrect.

[0113] Previously, segmented output blindly pursued rapid response without considering the correctness of the process, making it unsuitable for problem explanation. Thanks to the semantic block decoding format, this application can evaluate each step of the explanation, promptly correct process errors, and prevent the accumulation of errors into incorrect answers, balancing response speed and correctness, thus improving the accuracy of problem explanation. The evaluation model analyzes each semantic block after reading the problem; if errors are found, a corrected semantic block is output; otherwise, the original semantic block remains unchanged, and then the tool is used to present it to the user.

[0114] In some embodiments, step 104 includes: outputting semantic blocks one by one through an interactive video model based on valid question features and the answer content of the questions.

[0115] In some embodiments, the training process of the interactive video model includes:

[0116] Based on the effective question features and solutions of the sample questions, the interactive video model is used to infer G times to generate G complete explanation texts for the sample questions; where G is an integer greater than 1.

[0117] Input the G complete explanation texts of the sample questions into the evaluation model to obtain the evaluation results corresponding to each of the G complete explanation texts of the sample questions;

[0118] Based on the evaluation results corresponding to the G complete explanation texts of the sample questions, generate the corresponding rewards for the G complete explanation texts of the sample questions.

[0119] Based on the rewards corresponding to the G complete explanatory texts of the sample questions, an interactive video model is trained through group relative strategy optimization.

[0120] In the exemplary embodiment, the evaluation model is trained through a massive amount of correct and incorrect explanations, and thus possesses strong judgment and modification capabilities.

[0121] In an exemplary embodiment, if the evaluation result indicates that the complete explanation text is completely correct, the reward corresponding to the complete explanation text is set to 1 point; if the evaluation result indicates that the complete explanation text contains errors, the reward corresponding to the complete explanation text is set to 0 points.

[0122] In the exemplary embodiment, group relative policy optimization refers to GRPO (Group Relative Policy Optimization).

[0123] By cleverly incorporating the reward model from reinforcement learning into the evaluation model, which serves as a semantic block, common problems in language models, such as logical errors, fabricated facts, and inconsistencies, can be effectively addressed. This not only ensures correct answers but also demonstrates responsibility for the explanation process. The evaluation model originates from the reward model developed during reinforcement learning training. Through adversarial interaction with the interactive video model, the evaluation model is continuously optimized, improving the reliability of evaluation and correction.

[0124] In an exemplary embodiment, such as Figure 6 The diagram shown illustrates the reinforcement learning process of the evaluation model and the interactive video model. Figure 6 The first step is to train the evaluation model. A large number of correct and incorrect explanations are input into the evaluation model to obtain the evaluation results corresponding to the correct explanations: correct, the evaluation results corresponding to the incorrect explanations: incorrect, and the corrected explanations. The evaluation model is then trained. Figure 6The second step involves training the interactive video model. The question images and solutions for the sample questions are input into the interactive video model to obtain G complete explanation texts for each sample question. These G complete explanation texts are then input into the evaluation model to obtain the evaluation results for each of the G complete explanation texts. If the evaluation result indicates that the complete explanation text is completely correct, the reward for that complete explanation text is set to 1 point; if the evaluation result indicates that the complete explanation text contains errors, the reward for that complete explanation text is set to 0 points. Based on the rewards corresponding to the G complete explanation texts for each sample question, the interactive video model is trained using GRPO (Group Relative Policy Optimization).

[0125] In an exemplary embodiment, such as Figure 7 The image shown is a schematic diagram illustrating the process of generating the explanatory video. Figure 7 In this process, the question image and the answer to the question are input into the interactive video model, which then streams semantic blocks to respond to the user in real time.

[0126] For example, the content of the first semantic block is as follows:

[0127] <json>

[0128] {

[0129] "Type": "Reading the Question"

[0130] "Content": "Question 2: Given that angle 1 of a triangle is 35 degrees and angle 3 is 71 degrees, find the measure of angle 2."

[0131] }

[0132] < / json>

[0133] Paired identifiers <json> and< / json> The first semantic block has been confirmed to be fully decoded. The type of the first semantic block is "reading a question," and the content is sent to the speech synthesis tool.

[0134] For example, the content of the second semantic block is as follows:

[0135] <json>

[0136] {

[0137] "Type": "Topic"

[0138] "Content": "This question tests a geometric theorem: the sum of the angles of a triangle is 180 degrees."

[0139] "Display": "The sum of the interior angles of a triangle is 180 degrees."

[0140] "Mark": [

[0141] { "type": "highlight", "location": [12,15,201,339]},

[0142] { "type": "underline", "location": [130,281,201,339]}

[0143] ],

[0144] Question: What are the three angles of an isosceles right triangle in degrees?

[0145] }

[0146] < / json>

[0147] The second semantic block is input into the evaluation model, yielding: 1. Evaluation result: Error; 2. Error analysis: Inaccurate content description, lack of units in Display; 3. Decision: Correction. The corrected semantic block is obtained by correcting the second semantic block.

[0148] The content of the corrected semantic block obtained by correcting the second semantic block is as follows:

[0149] <json>

[0150] {

[0151] "Type": "Topic"

[0152] "Content": "This question tests a geometric theorem: the sum of the three interior angles of a triangle is 180 degrees."

[0153] "Display": "The sum of the interior angles of a triangle is 180°".

[0154] "Mark": [

[0155] { "type": "highlight", "location": [12,15,201,339]},

[0156] { "type": "underline", "location": [130,281,201,339]}

[0157] ],

[0158] Question: What are the three angles of an isosceles right triangle in degrees?

[0159] }

[0160] < / json>

[0161] The content and question in the corrected semantic block obtained by correcting the second semantic block are sent to the speech synthesis tool, and the whiteboard content and mark content are sent to the image rendering tool to finally obtain the explanation video.

[0162] In an exemplary embodiment, such as Figure 8The diagram shown is a schematic representation of a problem-solving method provided in an embodiment of this application. Figure 8 In the process, the question image is input into the answering model to obtain the question's explanation and answer; the question image, along with the explanation and answer, are input into the interactive video model to obtain semantic blocks; the semantic blocks include explanation content, diagram content, whiteboard content, and rhetorical questions; the question image, diagram content, and whiteboard content are rendered to obtain the explanation image; the explanation content is then processed into speech; if a user interrupts to ask a question, the current speech synthesis and image rendering operations are paused; the user's interrupted question is input into the question-answering model to obtain the response content, which is then processed into speech; the rhetorical questions are also processed into speech; the rhetorical questions and their corresponding user responses are input into the question-answering model to obtain the correctness judgment and explanation of the user's response text; the correctness judgment and explanation of the user's response text are then processed into speech; the interactive video model continuously outputs semantic blocks until all explanation steps for the question are completed. Figure 8 The two question-answering models in the text are the same question-answering model.

[0163] In summary, this application determines a compressed resolution that matches the resolution of the question image and encodes the compressed question image according to this resolution to obtain initial image features. This allows for dynamic adjustment of the compressed resolution based on the question image's resolution. Compared to existing technologies that encode the question image at a fixed resolution, this reduces invalid encoding and consequently reduces the amount of data in the initial image features. Cross-attention operations are performed on the initial image features and the initialized question features to obtain effective question features. The initialized question features are the initial values ​​of the effective question features. Effective question features include valid question information from the initial image features, allowing for the extraction of valid question information and eliminating interference from background information. Furthermore, the amount of data in the effective question features is less than that in the initial image features. Based on these effective question features, the solution content is generated. Since only the effective question features are used in generating the solution content, the amount of data from the image features involved in the solution is reduced, increasing the speed of generating the solution content and further improving the speed of question explanation.

[0164] The amount of data for effective question features is less than the amount of data for initial image features. Based on the effective question features and the answer content, semantic blocks are output one by one. Only effective question features are used in the generation of semantic blocks, which reduces the amount of data from image features involved in generating semantic blocks and improves the speed of semantic block generation. Furthermore, this application outputs semantic blocks one by one based on effective question features and the answer content. Each semantic block represents one step in the explanation of the question. When a semantic block is output, an explanation video is generated based on the output semantic block. This eliminates the need to wait for all explanation steps of the question to be generated before generating the explanation video. It allows the entire explanation process of the question to be broken down and output step by step, generating explanation videos, reducing user waiting time and improving the speed of question explanation.

[0165] Exemplary System

[0166] Optionally, embodiments of this application also provide a problem explanation system, such as... Figure 9 As shown, the problem explanation system includes:

[0167] Answer module 901 is used to determine a compressed resolution that matches the resolution of the question image, and to encode the question image after compression according to the compressed resolution to obtain initial image features; to perform cross-attention operation on the initial image features and the initialized question features to obtain effective question features; wherein, the initialized question features are the initial values ​​of the effective question features; the effective question features include the effective question information in the initial image features; and to generate the answer content of the question based on the effective question features.

[0168] The interactive video generation module 902 is used to output semantic blocks one by one based on the effective question features and the answer content of the question; wherein each semantic block is used to represent an explanation step of the question; and an explanation video is generated based on the output semantic block when each semantic block is output.

[0169] In an exemplary embodiment, the answer module 901 can be implemented through an answer model, by inputting the question image into the answer model to obtain the answer content of the question.

[0170] Optionally, the answer module 901 is specifically used to determine a downsampling ratio based on the resolution of the question image and the resolution of a preset region; wherein the preset region consists of multiple grids; downsampling the question image according to the downsampling ratio to obtain a downsampled image; selecting a target region in the preset region that matches the resolution of the downsampled image; wherein the target region consists of at least one of the grids; determining the resolution of the target region as the compressed resolution; adjusting the resolution of the downsampled image to the compressed resolution to obtain an adjusted image, and encoding the adjusted image to obtain initial image features.

[0171] In an exemplary embodiment, the interactive video generation module 902 can be implemented using an interactive video model and external tools (e.g., image rendering tools and speech synthesis tools). The question image and the answer to the question are input into the interactive video model to obtain various semantic blocks. By calling the speech synthesis tool and image rendering tool on the semantic blocks, an explanatory video is generated based on the output semantic blocks.

[0172] Optionally, the semantic block includes the explanation content, the marking content, and the blackboard writing content;

[0173] The interactive video generation module 902 is specifically used to synthesize speech from the explanatory content in the output semantic block to generate explanatory audio; to perform image rendering based on the marking content in the output semantic block to obtain marking results; wherein, the marking results are used to mark the question image; to perform image rendering based on the whiteboard content in the output semantic block to obtain a whiteboard image; to add the marking results and the whiteboard image to the question image to generate an explanatory image; and to generate an explanatory video based on the explanatory audio and the explanatory image.

[0174] Optionally, the semantic block may also include rhetorical questions;

[0175] The interactive video generation module 902 is specifically used to perform speech synthesis based on the rhetorical question content in the output semantic block to generate rhetorical question audio; and to generate an explanation video based on the explanation audio, the explanation image and the rhetorical question audio.

[0176] Optionally, the interactive video generation module 902 is specifically used to input the output semantic block into the evaluation model to obtain the evaluation result of the output semantic block; if the evaluation result indicates that the output semantic block is correct, generate an explanatory video based on the output semantic block; if the evaluation result indicates that the output semantic block is incorrect, correct the output semantic block to obtain a corrected semantic block, and generate an explanatory video based on the corrected semantic block.

[0177] Optionally, the interactive video generation module 902 is specifically used to output semantic blocks one by one through the interactive video model based on the effective question features and the answer content of the questions. The training process of the interactive video model includes: generating G complete explanation texts for the sample questions by inferring G times through the interactive video model based on the effective question features and the answer content of the sample questions; where G is an integer greater than 1; inputting the G complete explanation texts of the sample questions into the evaluation model to obtain the evaluation results corresponding to each of the G complete explanation texts of the sample questions; generating rewards corresponding to each of the G complete explanation texts of the sample questions based on the evaluation results corresponding to each of the G complete explanation texts of the sample questions; and training the interactive video model through group relative strategy optimization based on the rewards corresponding to each of the G complete explanation texts of the sample questions.

[0178] Optionally, the semantic block includes rhetorical questions, and the question explanation system also includes:

[0179] The question-and-answer module 903 is used to obtain the user's reply text corresponding to the question content, and based on the question content and the user's reply text corresponding to the question content, generate the correctness judgment result and explanation content of the user's reply text, and / or, if it is determined that a user interrupted the question, obtain the user interrupted the question content, and pause the operation of speech synthesis and image rendering on the output semantic block, and generate the reply content corresponding to the user interrupted the question content.

[0180] In an exemplary embodiment, the question-answering module 903 can be implemented through a question-answering model, by inputting the rhetorical question and the corresponding user reply text into the question-answering model to obtain the correctness judgment result and explanation content of the user reply text; and / or, by inputting the user interruption question into the question-answering model to obtain the reply content corresponding to the user interruption question.

[0181] The problem explanation system provided in this embodiment belongs to the same concept as the problem explanation method provided in the above embodiments of this application. It can execute the problem explanation method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the problem explanation method. Technical details not described in detail in this embodiment can be found in the specific processing content of the problem explanation method provided in the above embodiments of this application, and will not be repeated here.

[0182] The functions implemented by the above-mentioned question-answering module 901, interactive video generation module 902 and question-answering module 903 can be implemented by the same or different processors, and this application embodiment does not limit them.

[0183] Exemplary electronic devices

[0184] One embodiment of this application discloses an electronic device, see [link to relevant documentation] Figure 10 As shown, the device includes:

[0185] Memory 200 and processor 210;

[0186] The memory 200 is connected to the processor 210 and is used to store programs;

[0187] The processor 210 is used to implement the problem explanation method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0188] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0189] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:

[0190] A bus can include a pathway for transmitting information between various components of a computer system.

[0191] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0192] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0193] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0194] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0195] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0196] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0197] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement each step of any of the problem explanation methods provided in the above embodiments of this application.

[0198] Exemplary computer program products and storage media

[0199] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods described in the above embodiments of this specification according to various embodiments of this application.

[0200] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0201] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor using the steps described in the titles of various embodiments of this application above. Specifically, the following steps can be implemented:

[0202] Step 101: Determine the compressed resolution that matches the resolution of the question image, and encode the compressed question image according to the compressed resolution to obtain the initial image features.

[0203] Step 102: Perform cross-attention operation on the initial image features and the initial question features to obtain effective question features.

[0204] The initial question features are the initial values ​​of the valid question features; the valid question features include the valid question information in the initial image features.

[0205] Step 103: Based on the characteristics of valid questions, generate the answer content for the questions.

[0206] Step 104: Based on the valid question features and the answer content of the question, output semantic blocks one by one.

[0207] Each semantic block represents a step in explaining the question.

[0208] Step 105: For each semantic block output, generate an explanatory video based on the output semantic block.

[0209] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0210] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0211] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0212] The modules and sub-modules in the apparatus and terminal in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0213] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0214] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0215] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0216] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0217] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0218] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0219] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for explaining problems, characterized in that, include: The first ratio is obtained by dividing the length of the question image in pixels by the length of the preset region in pixels. The second ratio is obtained by dividing the width of the question image in pixels by the width of the preset region in pixels. The target ratio is rounded up to obtain the downsampling factor. The target ratio is the maximum value between the first ratio and the second ratio. The image of the question is downsampled according to the downsampling factor to obtain the downsampled image; The compressed resolution is determined based on the resolution of the downsampled image; The resolution of the downsampled image is adjusted to the compressed resolution to obtain the adjusted image, and the adjusted image is encoded to obtain the initial image features; A cross-attention operation is performed on the initial image features and the initial question features to obtain effective question features; wherein, the initial question features are the initial values ​​of the effective question features; the effective question features include the effective question information in the initial image features; Based on the aforementioned effective question characteristics, the answer content for the question is generated; Based on the valid question features and the answer content of the question, semantic blocks are output one by one; wherein each semantic block is used to represent an explanation step of the question; For each semantic block output, an explanatory video is generated based on the output semantic block.

2. The problem explanation method according to claim 1, characterized in that, The preset area is composed of multiple grids; Determining the compressed resolution based on the resolution of the downsampled image includes: Within the preset area, a target region matching the resolution of the downsampled image is selected; wherein, the target region consists of at least one of the grids; The resolution of the target region is determined as the compressed resolution.

3. The method for explaining problems according to claim 1, characterized in that, The semantic block includes the explanatory content, the marked content, and the blackboard writing content; The output-based semantic block generation of the explanatory video includes: The explanatory content in the output semantic block is synthesized into speech to generate an explanatory audio; Image rendering is performed based on the labeled content in the output semantic block to obtain the labeled result; wherein, the labeled result is used to label the question image; The whiteboard image is obtained by rendering the whiteboard content in the output semantic block. Add the marked image and the blackboard image to the title image to generate an explanation image; An explanatory video is generated based on the explanatory audio and the explanatory images.

4. The problem explanation method according to claim 3, characterized in that, The semantic block also includes rhetorical questions; The step of generating an explanatory video based on the explanatory audio and the explanatory images includes: Speech synthesis is performed based on the rhetorical question content in the output semantic block to generate rhetorical question audio; An explanatory video is generated based on the explanatory audio, the explanatory image, and the rhetorical question audio.

5. The problem-solving method according to claim 4, characterized in that, The method further includes: Obtain the user's response text corresponding to the question asked; Based on the rhetorical question and the corresponding user response text, a judgment result and explanation of the correctness of the user response text are generated.

6. The problem explanation method according to claim 3, characterized in that, The method further includes: If it is determined that a user interrupted the question, the content of the interrupted question is obtained, and the operation of speech synthesis and image rendering on the output semantic block is paused. Generate a response to the user's interrupted question.

7. The method for explaining problems according to claim 1, characterized in that, The output-based semantic block generation of the explanatory video includes: The output semantic block is input into the evaluation model to obtain the evaluation result of the output semantic block; If the evaluation result indicates that the output semantic block is correct, an explanatory video is generated based on the output semantic block; If the evaluation result indicates that the output semantic block is incorrect, the output semantic block is corrected to obtain a corrected semantic block, and an explanatory video is generated based on the corrected semantic block.

8. The method for explaining problems according to claim 7, characterized in that, Based on the valid question features and the answer content of the question, semantic blocks are output one by one, including: Based on the features of the effective questions and the answers to the questions, semantic blocks are output one by one through the interactive video model; The training process of the interactive video model includes: Based on the valid question features and the solution content of the sample questions, the interactive video model is used to infer G times to generate G complete explanation texts for the sample questions; where G is an integer greater than 1. Input the G complete explanation texts of the sample question into the evaluation model to obtain the evaluation results corresponding to each of the G complete explanation texts of the sample question; Based on the evaluation results corresponding to the G complete explanation texts of the sample questions, generate the corresponding rewards for the G complete explanation texts of the sample questions. Based on the rewards corresponding to the G complete explanatory texts of the sample questions, an interactive video model is trained through group relative strategy optimization.

9. A problem explanation system, characterized in that, include: The question-answering module is used to determine a first ratio obtained by dividing the length of the question image in pixels by the length of a preset region in pixels, and a second ratio obtained by dividing the width of the question image in pixels by the width of the preset region in pixels. The target ratio is then rounded up to obtain a downsampling factor; the target ratio is the maximum of the first and second ratios. Based on the downsampling factor, the question image is downsampled to obtain a downsampled image. A compressed resolution is determined based on the resolution of the downsampled image. The resolution of the downsampled image is adjusted to the compressed resolution to obtain an adjusted image, and the adjusted image is encoded to obtain initial image features. Cross-attention operations are performed on the initial image features and the initialized question features to obtain effective question features; the initialized question features are the initial values ​​of the effective question features. The effective question features include valid question information from the initial image features. Based on the effective question features, the answer content for the question is generated. An interactive video generation module is used to output semantic blocks one by one based on the valid question features and the answer content of the question; wherein each semantic block is used to represent an explanation step of the question; and an explanation video is generated based on the output semantic block when each semantic block is output.

10. The system according to claim 9, characterized in that, The semantic block includes rhetorical questions, and the system also includes: The question-and-answer module is used to obtain the user's response text corresponding to the question, and based on the question and the user's response text, generate the correctness judgment result and explanation content of the user's response text, and / or, if it is determined that a user interrupted the question, obtain the user interruption question content, pause the operation of speech synthesis and image rendering on the output semantic block, and generate the reply content corresponding to the user interruption question content.

11. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the problem explanation method as described in any one of claims 1 to 8 by running the program in the memory.

12. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to perform the problem explanation method as described in any one of claims 1 to 8.

Images