A software system development method based on a multi-modal AI large model

Abstract

The application relates to the technical field of data processing, in particular to a software system development method based on a multi-modal AI large model, which comprises the following steps: S1, acquiring multi-modal data; S2, pre-processing the multi-modal data and acquiring a fusion modal data block; S3, data layering of the fusion modal data block; S4, bias adjustment of the data layering process and construction of a memory atlas; and S5, output of a working software system. According to the application, multi-modal data is processed, results are adjusted, bias is updated, and differences are optimized, so that data containing mixed text, pictures, videos and audio can be processed at the same time, data bias can be removed, the demand of a target user for the working software system can be updated in real time, and therefore the accuracy and efficiency of robot application software development are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a software system development method based on a multi-modal AI large model. BACKGROUND

[0002] Robots cannot generate work software systems in real time according to user needs when adapting to various work scenarios, cannot realize open application, have application limitations, have long-term dependency problems of long context windows, and data bias problems, and the long-term dependency problems of long context windows can cause the multi-modal AI large model computing power in the robot application software to be limited when a user inputs long resources and multiple rounds of dialogues, and key details can be missing when context is associated, and the long-term dependency problem cannot be formed.

[0003] Chinese Patent Publication No. CN117420760A discloses a multi-modal control algorithm fusion method suitable for robot autonomous cooperation, sets an algorithm layer to form a large algorithm library to provide algorithm support for various work of the robot, sets a task layer to translate work tasks given by humans into a set of executable tasks for the robot through big data and AI classification methods, connects similar task analysis methods by importing target tasks, and splits the target tasks into N specific robot executable task points or steps through built-in reasoning, analysis and other algorithms; a matching layer is set to deploy a special matching algorithm, a matching platform, and a matching interface to connect and complete the matching of tasks and algorithms; a decision layer is set to provide decision-making for the matching results, and control learning algorithms and model algorithms responsible for common sense, decision-making, and reasoning; it can be seen that this scheme still has the problems of being unable to generate work software systems in real time according to user needs, having long-term dependency and data bias problems of long context windows. SUMMARY

[0004] Therefore, the present application provides a software system development method based on a multi-modal AI large model to overcome the problems of being unable to generate work software systems in real time according to user needs, having data bias, and low robot application software development efficiency in the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides a software system development method based on a multi-modal AI large model, which comprises:

[0006] Step S1, acquiring multi-modal data;

[0007] Step S2, pre-processing the multi-modal data according to a preprocessing method to obtain processed multi-modal data, and acquiring a fusion modal data block according to the processed multi-modal data;

[0008] Step S3, data layering is performed on the fusion modal data block to obtain a result data block summary, anchor scanning is performed on the fusion modal data block to obtain key entities, and result adjustment is performed on the result data block summary according to the key entities, and result update is performed on the result adjustment according to the destructive scenario;

[0009] Step S4, bias adjustment is performed on the data layering process according to a bias coefficient, bias update is performed on the bias adjustment process according to an entity strictness scenario, and a memory graph is constructed according to the result data block summary through a memory graph construction method to obtain a memory graph;

[0010] Step S5, the working software system is output according to the memory graph and the multi-modal data, the output process of the working software system is corrected according to real-time software generation requirements, and the memory graph and the data layering are optimized according to a difference coefficient;

[0011] The destructive scenario refers to a preset text database for identifying destructive texts in the fusion modal data block, and the destructive text refers to an unconventional text instance that may cause analysis errors.

[0012] The key entity scenario is compared with a strictness scenario library, the existence of the key entity scenario in the strictness scenario library is judged according to the comparison result, and the entity strictness scenario is output according to the judgment result, and the strictness scenario library refers to a preset scenario text data set for judging the entity strictness scenario.

[0013] Further, the multi-modal data is preprocessed according to a preprocessing method, and the preprocessing method includes:

[0014] Step A01, text cleaning is performed on a target text in the multi-modal data to obtain a processed target text;

[0015] Step A02, key frame extraction is performed on a target video in the multi-modal data to obtain target key frames, and motion feature extraction is performed on the target key frames to obtain a processed target video;

[0016] Step A03, noise reduction processing is performed on a target voice in the multi-modal data to obtain a noise-reduced target voice, and voice feature extraction is performed on the noise-reduced target voice to obtain a processed target voice;

[0017] Step A04, format unification is performed on a target picture in the multi-modal data to obtain a unified target picture, and normalization processing is performed on the unified target picture to obtain a processed target picture;

[0018] Step A05, the processed target text, the processed target video, the processed target voice, the processed target picture, and real-time software generation requirements are taken as processed multi-modal data.

[0019] Further, in step S2, when acquiring the fused modal data block based on the processed multimodal data, the processed multimodal data is divided into data blocks using a data block method, which includes:

[0020] Step B01: Perform text conversion on the processed target speech to obtain the second target text; perform length block division on the second target text and the processed target text to obtain text modality blocks.

[0021] Step B02: Perform keyframe segmentation on the processed target video to obtain video modal segments;

[0022] Step B03: Divide the processed target image into grid blocks to obtain image modal blocks;

[0023] Step B04: The text modal blocks, video modal blocks, and image modal blocks are used as multimodal data blocks.

[0024] Further, in step S2, when acquiring the fused modal data block based on the processed multimodal data, the multimodal data block is modally converted according to a modal conversion method, the modal conversion method including:

[0025] Step S21: Input the video modal blocks into the target detection model to obtain video text blocks, and add timestamps to the video text blocks to obtain target video text blocks;

[0026] Step S22: Input the image modal blocks into the visual language model to obtain the target image text blocks;

[0027] Step S23: The text modal blocks, target video text blocks, and target image text blocks are used as fused modal data blocks.

[0028] Further, in step S3, when performing data layering on the fused modal data block, the fused modal data block is layered according to a data layering method, which includes:

[0029] Step S31: Set the number of viewpoints n0. Initialize the definition of the fused modal data blocks according to the number of viewpoints n0 and the analysis viewpoints to obtain the data block summary nf={nf1,nf2,nf3,...,nfi}, where nf1 is the first feature summary set in each data block summary, set nf1={nf11,nf12,nf13,...,nf1h1}, nf2 is the second feature summary set in each data block summary, set nf2={nf21,nf22,nf23,...,nf2h2}, nf3 is the third feature summary set in each data block summary, set nf3={nf31,nf32,nf33,...,nf3h3},...,nfi is the i-th feature summary set in each data block summary, set nfi={nfi1,nfi2,nfi3,...,nfihi};

[0030] Step S32: Obtain the first viewpoint confidence score nh1={nh11,nh12,nh13,...,nh1h1} based on the first feature summary set nf1={nf11,nh12,nh13,...,nh1h1};

[0031] The confidence scores nh2={nh21,nh22,nh23,...,nh2h2} of the second perspective are obtained based on the second feature summary set nf2={nf21,nh22,nh23,...,nh2h2}.

[0032] The confidence scores of the third perspective nh3={nh31,nh32,nh33,...,nh3h3} are obtained based on the third feature summary set nf3={nf31,nh32,nh33,...,nh3h3}.

[0033] ...

[0034] The confidence scores nhi = {nhi1, nhi2, nhi3, ..., nhih3} of the i-th viewpoint are obtained based on the i-th feature summary set nfi = {nfi1, nhi2, nhi3, ..., nhih3}.

[0035] Step S33: Calculate the first confidence level Ba1 based on the first perspective confidence level nh1={nh11,nh12,nh13,...,nh1h1}, the total number of first feature summary results n1, and the first perspective confidence level weight β1={β11, β12, β13,...,β1h1}, and set... The first confidence level Ba1 is obtained;

[0036] The confidence level Ba2 is calculated based on the second-view confidence level nh2={nh21,nh22,nh23,...,nh2h2}, the total number of second feature summary results n2, and the second-view confidence level weight β2={β21, β22, β23,...,β2h2}. The second confidence level, Ba2, is obtained.

[0037] The confidence level Ba3 is calculated based on the third-view confidence level nh3={nh31,nh32,nh33,...,nh3h3}, the total number of third-feature summary results n3, and the third-view confidence weight β3={β31,β32,β33,...,β3h3}. The third confidence level, Ba3, is obtained.

[0038] ...

[0039] The confidence score Bai is calculated based on the confidence score nhi={nhi1,nhi2,nhi3,...,nhih3} of the i-th perspective, the total number of i-th feature summary results ni, and the confidence score weight βi={βi1,βi2,βi3,...,βihi} of the i-th perspective. We obtain the confidence level Bai of the i-th term;

[0040] Step S34: Obtain the maximum confidence Bax based on the first confidence level Ba1, the second confidence level Ba2, the third confidence level Ba3, ..., the i-th confidence level Bai;

[0041] Step S35: Compare the maximum confidence level Bax with the preset confidence level Bax0. Based on the comparison result, determine whether the maximum confidence level meets the standard, and output the result data block summary based on the determination result.

[0042] When Bax≥Bax0, the maximum confidence level is considered met, and the feature summary set corresponding to the maximum confidence level is output as the result data block summary.

[0043] When Bax < Bax0, the maximum confidence level is deemed unqualified. The maximum confidence level is then reviewed to obtain the reviewed maximum confidence level. The manually input feature summary set is then output as the result data block summary.

[0044] In step S3, when anchoring the fused modal data block, the fused modal data block is input into the key scanning model to obtain the key entity.

[0045] Further, in step S3, when adjusting the result data block summary based on key entities, the entity proportion coefficient Y is obtained based on the key entities, the entity proportion coefficient Y is compared with the preset entity proportion coefficient Y0, the compliance status of the entity proportion coefficient is judged based on the comparison result, and the preset reliability Bax0 is adjusted based on the judgment result, wherein:

[0046] When Y≥Y0, the entity proportion coefficient is deemed to have met the standard, and no adjustment is made to the pre-set confidence level Bax0.

[0047] When Y < Y0, the entity proportion coefficient is deemed unqualified. The preset reliability Bax0 is adjusted based on the adjustment coefficient jg, where jg = 1.57 - 0.23 × e -0.38×(Y0-Y) Where e is the base of the natural logarithm, the adjusted preset confidence level Bax0` is obtained, Bax0` is set to Bax0×jg, the preset confidence level Bax0 is replaced with the adjusted preset confidence level Bax0`, and the maximum confidence level Bax is recompared with the adjusted preset confidence level Bax0`.

[0048] The entity proportion coefficient refers to a value used to measure the proportion of key entities in the fused modal data block. This embodiment does not limit the specific method of obtaining the entity proportion coefficient. Those skilled in the art can freely choose according to actual needs. For example, if the total number of text data in the fused modal data block is set to wb and the total number of key entities in the text data is set to wg, then the entity proportion coefficient Y = wg / wb.

[0049] In step S3, when updating the results based on the destructive scenario, an initial destructive scenario is set up to obtain an initial destructive scenario. The fused modal data block is compared with the initial destructive scenario. Based on the comparison result, the existence of the fused modal data block in the initial destructive scenario is determined, and the key entities are updated based on the determination result, wherein:

[0050] When the existence of the fused modal data block is not found in the initial destructive scenario, no entity update is performed on the key entity;

[0051] When the existence of the fused modal data block is confirmed in the initial destructive scenario, the key entities are updated.

[0052] Specifically, when updating key entities, destructive text is removed to obtain a standard fusion modality data block, which is then re-inputted into the key scanning model to obtain the updated key entity.

[0053] Furthermore, in step S4, when adjusting the bias during the process of data stratification based on the bias coefficient, the bias is adjusted according to the probability of the fused modal data block type. ={ , , ,..., } and total number of categories Bias coefficient Perform calculations and set Obtain the bias coefficient , bias coefficient The bias coefficient is compared with the preset bias coefficient Pmg0. Based on the comparison results, the compliance of the bias coefficient is judged, and the bias of the number of viewpoints n0 is adjusted according to the judgment results.

[0054] when When the value is ≥Pmg0, step S4 determines that the bias coefficient meets the standard and does not adjust the number of viewpoints n0 for bias.

[0055] when When <Pmg0, step S4 determines that the bias coefficient does not meet the standard, and adjusts the bias of the number of viewpoints n0 to obtain the adjusted number of viewpoints n0`. Set n0` = n0 + 1, and re-layer the data and adjust the bias according to the adjusted number of viewpoints n0` and the newly added analysis viewpoints until step S4 determines that the bias coefficient meets the standard.

[0056] Further, in step S4, during the bias update process of bias adjustment based on entity strict scenarios, the key entity scenario is compared with the strict scenario library. Based on the comparison result, the existence of the key entity scenario in the strict scenario library is determined, and the entity strict scenario is output based on the determination result. The preset bias coefficient Pmg0 is also updated based on the determination result, wherein:

[0057] When step S4 determines that the existence of the key entity scene in the strict scene library is non-existent, the entity strict scene is not output, and the preset bias coefficient P0 is not updated.

[0058] When step S4 determines that the existence of the key entity scene in the strict scene library is true, the entity strict scene is output and the preset bias coefficient Pmg0 is updated.

[0059] Specifically, when updating the preset bias coefficient Pmg0, the preset bias coefficient Pmg0 is updated according to the bias update coefficient pg to obtain the updated preset bias coefficient Pmg0`. Pmg0` is set to Pmg0×pg. The preset bias coefficient Pmg0 is replaced with the updated preset bias coefficient Pmg0`, and the bias coefficient Pmg is re-compared with the updated preset bias coefficient Pmg0`.

[0060] Further, in step S4, a memory map is constructed based on the result data block summary using a memory map construction method, the memory map construction method including:

[0061] Step S41: Construct the knowledge base to obtain the basic knowledge base;

[0062] Step S42: Compare the result data block summary with the basic knowledge base to obtain a preliminary memory map;

[0063] Step S42: Based on the repair coefficient xb, perform contradiction repair on the preliminary memory map to obtain the memory map.

[0064] Furthermore, in step S5, the working software system is output based on the memory map and multimodal data;

[0065] In step S5, when correcting the output process of the working software system according to the real-time software generation requirements, the real-time software generation requirement ts is compared with the previous real-time software generation requirement ts0. Based on the comparison result, the update status of the multimodal data is judged, and the multimodal data is corrected based on the judgment result. Specifically:

[0066] If the update status of the multimodal data is determined to be no update, no working correction is performed on the multimodal data;

[0067] When the update status of the multimodal data is determined to be updated, the multimodal data is corrected to obtain corrected multimodal data. Steps S2 to S5 are repeated based on the corrected multimodal data to obtain the corrected working software system. The corrected working software system is then output as the working software system.

[0068] In step S5, when performing difference optimization on the memory map and data stratification based on the difference coefficient, if the update status of the multimodal data is "updated", the difference coefficient CY is calculated based on the time step nfd, the update difference coefficient qy, the working software system similarity rf, the time step weight nm1, the update difference coefficient weight nm2, and the working software system similarity weight nm3. The result is set as CY = nfd × nm1 + qy × nm2 + rf × nm3. The difference coefficient CY is then compared with the first preset difference coefficient CY1 and the second preset difference coefficient CY2. Based on the comparison result, the difference in the difference coefficient is judged, and based on the judgment result, the repair coefficient xb and the number of viewpoints n0 are optimized.

[0069] When CY≤CY1, the difference in the difference coefficient is determined to be a low degree of deviation, and no difference optimization is performed on the repair coefficient xb and the number of viewpoints n0.

[0070] When CY1 < CY ≤ CY2, the difference in the difference coefficient is determined to be a moderate deviation. The repair coefficient xb is then optimized based on the first difference coefficient dy, with dy = 1.28 - 0.22 × e. -0.48×(CY-CY1) Where e is the base of the natural logarithm, the optimized repair coefficient xb` is obtained, xb` is set to xb×dy, the repair coefficient xb is replaced with the optimized repair coefficient xb`, and the contradiction is repaired again on the preliminary memory map according to the optimized repair coefficient xb`;

[0071] When CY > CY2, the difference in the difference coefficient is determined to be a height deviation. Difference optimization is then performed on the number of viewing angles n0, based on the second difference coefficient df. df is set to 1.61 - 0.28 × e. -0.66×(CY-CY2) Where e is the base of the natural logarithm, the optimized number of viewpoints n0`` is obtained, and n0`` is set to n0×df. The value of n0`` is rounded to a positive integer. The number of viewpoints n0 is replaced with the optimized number of viewpoints n0``, and the fused modal data block is redefined based on the optimized number of viewpoints n0`` and the newly added analysis viewpoints.

[0072] Compared with existing technologies, the beneficial effects of this invention are as follows: the software system development method based on a multimodal AI large model acquires multimodal data in step S1, so as to output a working software system suitable for the target user based on the multimodal data, thereby improving the efficiency of the robot in generating the working software system. The method also acquires fused modal data blocks in step S2, unifying the format of the multimodal data and converting it into a fixed vector to improve the accuracy and applicability of the multimodal data. Furthermore, the method acquires a summary of the result data blocks in step S3, thereby reducing... Low data heterogeneity enables unified data representation and improves the efficiency of feature extraction from multimodal data, thereby focusing on key information and enhancing the effectiveness of multimodal data. The software system development method based on the multimodal AI large model also adjusts bias in step S4 to avoid the impact of logical defects in key entity scenarios on the achievement of the bias coefficient, thereby improving the accuracy of the output working software system and avoiding the omission of key details in contextual relationships. The software system development method based on the multimodal AI large model also performs working correction on the output working software system in step S5 to update the needs of target users in real time, thereby improving the efficiency of robot application software development. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the software system development method based on a multimodal AI large model in this embodiment. Detailed Implementation

[0074] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0075] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0076] Please see Figure 1 As shown, this is a structural diagram of the software system development method based on a multimodal AI large model in this embodiment. The method includes:

[0077] Step S1: Acquire multimodal data;

[0078] Step S2: Preprocess the multimodal data according to the preprocessing method to obtain processed multimodal data, and also obtain the fused modal data block based on the processed multimodal data;

[0079] Step S3: Perform data layering on the fused modal data block to obtain a result data block summary; perform anchor scanning on the fused modal data block to obtain key entities; adjust the result data block summary based on the key entities; and update the result adjustment based on the destructive scenario.

[0080] Step S4 involves adjusting the bias of the data stratification process based on the bias coefficient, updating the bias of the bias adjustment process based on the strict scenario of the entity, and constructing the memory graph based on the result data block summary using the memory graph construction method to obtain the memory graph.

[0081] Step S5: Output the working software system based on the memory map and multimodal data, and make working corrections to the output process of the working software system according to the real-time software generation requirements. Then, optimize the memory map and data stratification based on the difference coefficient.

[0082] Specifically, the software system development method based on a multimodal AI large model is applied in the development of robot software systems. It processes multimodal data, adjusts results, updates biases, and optimizes differences to simultaneously handle mixed text, image, video, and audio data, remove data biases, and update the target user's requirements for the working software system in real time. Specifically, step S1 acquires multimodal data to facilitate the subsequent output of a working software system suitable for the target user, thereby improving the efficiency of robot-generated working software systems. Step S2 further acquires fused modal data blocks, unifies the format of the multimodal data, and converts it into a fixed vector to improve the accuracy of the multimodal data. To ensure accuracy and applicability, the software system development method based on a multimodal AI model further obtains a summary of the result data blocks in step S3 to reduce data heterogeneity, unify the data representation, and improve the efficiency of feature extraction from multimodal data, thereby focusing on key information and enhancing the effectiveness of multimodal data. The method also adjusts bias in step S4 to avoid logical defects in key entity scenarios affecting the achievement of the bias coefficient, thus improving the accuracy of the output working software system and avoiding omission of key details in contextual relationships. Finally, the method corrects the output working software system in step S5 to update the target user's needs in real time, thereby improving the accuracy and efficiency of the robot-generated working software system.

[0083] Specifically, in step S1, multimodal data is acquired.

[0084] Specifically, in this embodiment, the data input by the target user on the robot's interactive interface is used as multimodal data. This multimodal data includes target text, target video, target speech, target image, and real-time software generation requirements. The target text refers to the key characters input by the target user on the robot's interactive interface to control commands. The target video refers to the video file input by the target user on the robot's interactive interface to convey commands. The target speech refers to the speech input by the target user on the robot's interactive interface to convey commands. The target image refers to the image input by the target user on the robot's interactive interface to convey commands. The real-time software generation requirements refer to the requirements for adjusting software functions, service logic, and resource allocation generated by the system in real-time during the operation of the robot software system, based on real-time acquired data and real-time input data from the target user. The target user refers to the person using the robot's output software system.

[0085] Specifically, in step S1, multimodal data is acquired so that a working software system suitable for the target user can be output based on the multimodal data, thereby improving the efficiency of the robot in generating the working software system.

[0086] Specifically, in step S2, the multimodal data is preprocessed according to a preprocessing method, which includes:

[0087] Step A01: Perform text cleaning on the target text in the multimodal data to obtain the processed target text;

[0088] Step A02: Extract keyframes from the target video in the multimodal data to obtain target keyframes, and extract motion features from the target keyframes to obtain the processed target video.

[0089] Step A03: Denoise the target speech in the multimodal data to obtain the denoised target speech, and extract speech features from the denoised target speech to obtain the processed target speech.

[0090] Step A04: Unify the format of the target image in the multimodal data to obtain the unified target image, and normalize the unified target image to obtain the processed target image;

[0091] Step A05: The processed target text, processed target video, processed target audio, processed target image, and real-time software-generated requirements are taken as processed multimodal data.

[0092] Specifically, text cleaning refers to the process of converting target text into a fixed-dimensional vector representation. This embodiment does not limit the specific method of text cleaning; those skilled in the art can freely choose according to actual needs, such as software processing. Keyframe extraction refers to the process of extracting keyframes from the target video to reduce the data volume. This embodiment does not limit the specific method of keyframe extraction; those skilled in the art can freely choose according to actual needs, such as using an adaptive sampling strategy. Motion feature extraction refers to the process of converting the motion features of the target keyframes into a vector representation. This embodiment does not limit the specific method of motion feature extraction; those skilled in the art can freely choose according to actual needs, such as optical flow algorithms. Noise reduction refers to the process of removing background noise from the target speech. This embodiment does not limit the specific method of noise reduction; those skilled in the art can freely choose according to actual needs, such as Wiener filtering. Speech feature extraction refers to the process of converting the denoised target speech into a vector representation and extracting semantic features. This embodiment does not limit the method of speech feature extraction. Those skilled in the art can freely choose according to actual needs. For example, the denoised target speech can be converted into a Mel spectrogram, and semantic features can be extracted through a speech recognition feature model. The Mel spectrogram refers to a frequency domain representation method of speech signals based on the characteristics of human hearing. The speech recognition feature model refers to a convolutional neural network model that takes the Mel spectrogram of the target speech as input data and semantic features as output data. This embodiment does not limit the specific construction method of the speech recognition feature model. Those skilled in the art can freely choose according to actual needs. For example, the Mel spectrogram of the historical target speech and its corresponding semantic features can be used as a training set to train the speech recognition feature model to obtain the speech recognition feature model. The format unification refers to the process of unifying the target image into the same format and size. This embodiment does not limit the specific method of format unification. Those skilled in the art can freely choose according to actual needs, such as software processing. The normalization processing refers to the process of scaling and projecting the unified target image into the [0, 1] interval. This embodiment does not limit the specific method of normalization processing. Those skilled in the art can freely choose according to actual needs, such as normalization processing through Pathon.

[0093] Specifically, in step S2, the multimodal data is preprocessed to convert the target text, target video, and target speech in the multimodal data into fixed vectors. At the same time, the target image is mapped to a suitable interval through mathematical transformation, so that the working software system can be output based on the processed multimodal data, thereby improving the accuracy and applicability of the output working software system.

[0094] Specifically, in step S2, when acquiring the fused modal data block based on the processed multimodal data, the processed multimodal data is divided into data blocks using a data segmentation method, which includes:

[0095] Step B01: Perform text conversion on the processed target speech to obtain the second target text; perform length block division on the second target text and the processed target text to obtain text modality blocks.

[0096] Step B02: Perform keyframe segmentation on the processed target video to obtain video modal segments;

[0097] Step B03: Divide the processed target image into grid blocks to obtain image modal blocks;

[0098] Step B04: The text modal blocks, video modal blocks, and image modal blocks are used as multimodal data blocks.

[0099] Specifically, the text conversion refers to the process of converting the language content expressed in the processed target speech into text. This embodiment does not limit the specific method of text conversion; those skilled in the art can freely choose according to actual needs, such as using automatic speech recognition to convert the processed target speech into text. The length segmentation refers to the process of segmenting the characters in the second target text and the processed target text into blocks according to a fixed number of characters. The fixed number of characters refers to a preset character segmentation standard value for length segmentation. This embodiment does not limit the specific value of the fixed number of characters; for example, setting the fixed number of characters to 500 characters per text modality block. This embodiment does not limit the specific method of length segmentation; those skilled in the art can freely choose according to actual needs, such as software processing. The keyframe segmentation refers to dividing the processed target video into frames. In the process of data block division, this embodiment does not limit the specific method of keyframe block division. For example, the frame sequence in the processed target video is divided into data blocks according to a preset number of blocks. The preset number of blocks refers to the number of frames contained in a single video modal block when dividing the frame sequence in the processed target video. For example, if the total number of frames in the processed target video is N and the preset number of blocks is n1 frames, then the total number of video modal blocks Np is Np = N / n1. The grid block division refers to the process of dividing the processed target image into image modal blocks. This embodiment does not limit the specific method of grid block division. Those skilled in the art can freely choose according to actual needs. For example, the processed target image is divided into a grid of fixed size, and each grid is an image modal block. This embodiment does not limit the fixed size of the grid. For example, the fixed size of the grid is set to an 8×8 pixel grid.

[0100] Specifically, in step S2, the processed multimodal data is divided into data blocks to reduce data processing complexity, reduce the data scale processed by the system in a single operation, reduce computational consumption, and thereby improve the robot's efficiency in utilizing multimodal data.

[0101] Specifically, in step S2, when acquiring the fused modal data block based on the processed multimodal data, the multimodal data block is modally converted according to a modal conversion method, the modal conversion method including:

[0102] Step S21: Input the video modal blocks into the target detection model to obtain video text blocks, and add timestamps to the video text blocks to obtain target video text blocks;

[0103] Step S22: Input the image modal blocks into the visual language model to obtain the target image text blocks;

[0104] Step S23: The text modal blocks, target video text blocks, and target image text blocks are used as fused modal data blocks.

[0105] Specifically, the target detection model refers to a convolutional neural network model that takes video modal blocks as input data and video text blocks as output data. This embodiment does not limit the specific construction method of the target detection model. Those skilled in the art can freely choose according to actual needs. For example, the target detection model can be trained by using historical video modal blocks and their corresponding video text blocks as training data. The timestamp annotation refers to the process of annotating the video time period corresponding to each video text block. This embodiment does not limit the specific method of timestamp annotation. Those skilled in the art can freely choose according to actual needs. For example, it can be annotated using video editing software. The visual language model refers to a convolutional neural network model that takes image modal blocks as input data and target image text blocks as output data. This embodiment does not limit the specific construction method of the visual language model. Those skilled in the art can freely choose according to actual needs. For example, the visual language model can be trained by using historical image modal blocks and their corresponding target image text blocks as training data.

[0106] Specifically, in step S2, the multimodal data blocks are modally converted to a unified text format fused modal data block, which facilitates subsequent data layering, saves computing resources, and improves the efficiency of outputting to the working software system.

[0107] Specifically, in step S3, when performing data layering on the fused modal data block, the fused modal data block is layered according to a data layering method, which includes:

[0108] Step S31: Set the number of viewpoints n0. Initialize the definition of the fused modal data blocks according to the number of viewpoints n0 and the analysis viewpoints to obtain the data block summary nf={nf1,nf2,nf3,...,nfi}, where nf1 is the first feature summary set in each data block summary, set nf1={nf11,nf12,nf13,...,nf1h1}, nf2 is the second feature summary set in each data block summary, set nf2={nf21,nf22,nf23,...,nf2h2}, nf3 is the third feature summary set in each data block summary, set nf3={nf31,nf32,nf33,...,nf3h3},...,nfi is the i-th feature summary set in each data block summary, set nfi={nfi1,nfi2,nfi3,...,nfihi};

[0109] Step S32: Obtain the first viewpoint confidence score nh1={nh11,nh12,nh13,...,nh1h1} based on the first feature summary set nf1={nf11,nh12,nh13,...,nh1h1};

[0110] The confidence scores nh2={nh21,nh22,nh23,...,nh2h2} of the second perspective are obtained based on the second feature summary set nf2={nf21,nh22,nh23,...,nh2h2}.

[0111] The confidence scores of the third perspective nh3={nh31,nh32,nh33,...,nh3h3} are obtained based on the third feature summary set nf3={nf31,nh32,nh33,...,nh3h3}.

[0112] ...

[0113] The confidence scores nhi = {nhi1, nhi2, nhi3, ..., nhih3} of the i-th viewpoint are obtained based on the i-th feature summary set nfi = {nfi1, nhi2, nhi3, ..., nhih3}.

[0114] Step S33: Calculate the first confidence level Ba1 based on the first perspective confidence level nh1={nh11,nh12,nh13,...,nh1h1}, the total number of first feature summary results n1, and the first perspective confidence level weight β1={β11, β12, β13,...,β1h1}, and set... The first confidence level Ba1 is obtained;

[0115] The confidence level Ba2 is calculated based on the second-view confidence level nh2={nh21,nh22,nh23,...,nh2h2}, the total number of second feature summary results n2, and the second-view confidence level weight β2={β21, β22, β23,...,β2h2}. The second confidence level, Ba2, is obtained.

[0116] The confidence level Ba3 is calculated based on the third-view confidence level nh3={nh31,nh32,nh33,...,nh3h3}, the total number of third-feature summary results n3, and the third-view confidence weight β3={β31,β32,β33,...,β3h3}. The third confidence level, Ba3, is obtained.

[0117] ...

[0118] The confidence score Bai is calculated based on the confidence score nhi={nhi1,nhi2,nhi3,...,nhih3} of the i-th perspective, the total number of i-th feature summary results ni, and the confidence score weight βi={βi1,βi2,βi3,...,βihi} of the i-th perspective. We obtain the confidence level Bai of the i-th term;

[0119] Step S34: Obtain the maximum confidence Bax based on the first confidence level Ba1, the second confidence level Ba2, the third confidence level Ba3, ..., the i-th confidence level Bai;

[0120] Step S35: Compare the maximum confidence level Bax with the preset confidence level Bax0. Based on the comparison result, determine whether the maximum confidence level meets the standard, and output the result data block summary based on the determination result.

[0121] When Bax≥Bax0, the maximum confidence level is considered met, and the feature summary set corresponding to the maximum confidence level is output as the result data block summary.

[0122] When Bax < Bax0, the maximum confidence level is deemed unqualified. The maximum confidence level is then reviewed to obtain the reviewed maximum confidence level. The manually input feature summary set is then output as the result data block summary.

[0123] Specifically, the number of perspectives refers to the number of different modal types into which the fused modal data block is divided based on multiple analysis dimensions. This embodiment does not limit the number of perspectives; those skilled in the art can freely choose according to actual needs, such as setting the number of perspectives n0=3. The analysis perspective refers to the definition and division criteria set when initializing the fused modal data block. This embodiment does not limit the analysis perspective; those skilled in the art can freely choose according to actual needs, such as using lexical and syntactic features and text structure features as definition and division criteria to initialize the fused modal data block. The initial definition is... This refers to the process of dividing the fused modal data block into n0 different modal types based on the number of perspectives, and then defining features for each modal type according to the analysis perspective to obtain a summary of each data block. This embodiment does not limit the specific method of initialization definition; those skilled in the art can freely choose according to actual needs, such as using the K-means clustering algorithm for initialization definition. The data block summary nf={nf1,nf2,nf3,...,nfi} refers to the set of feature summary results obtained for the same fused data block under n0 different analysis perspectives, where i is the order of the target data block summaries, i is a positive integer, and the most significant difference between i and i is a positive integer. The maximum value is n0. The first feature summary set nf1={nf11,nf12,nf13,...,nf1h1} is the set of feature summary results obtained from the first analysis perspective in each data block summary, where nf11 is the first feature summary result in the first feature summary set, nf12 is the second feature summary result in the first feature summary set, nf13 is the third feature summary result in the first feature summary set, nf1h1 is the h1-th feature summary result in the first feature summary set, h1 is the order of the feature summary results, and h1 is a positive integer. The second feature summary set nf2={nf2 {nf21, nf22, nf23, ..., nf2h2} represents the set of feature summary results obtained from the second analytical perspective in each data block summary. Here, nf21 is the first feature summary result in the second feature summary set, nf22 is the second feature summary result in the second feature summary set, nf23 is the third feature summary result in the third feature summary set, and nf2h2 is the h2-th feature summary result in the second feature summary set, where h2 is the order of the feature summary results in the second feature summary set and is a positive integer. The third feature summary set nf3 = {nf31, nf32, nf33, ...{nf3h3} represents the set of feature summary results obtained from the third analytical perspective in each data block summary, where nf31 is the first feature summary result in the third feature summary set, nf32 is the second feature summary result in the third feature summary set, nf33 is the third feature summary result in the third feature summary set, nf3h3 is the h3th feature summary result in the third feature summary set, h3 is the order of the feature summary results in the third feature summary set, and h3 is a positive integer. The i-th feature summary set nfi={nfi1,nfi2,nfi3,...,nfihi} is the set of feature summary results obtained from the i-th analytical perspective in each data block summary, where nfi1 is... Let nfi1 be the first feature summary result in the i-th feature summary set, nfi2 be the second feature summary result in the i-th feature summary set, nfi3 be the third feature summary result in the i-th feature summary set, and nfihi be the hi-th feature summary result in the i-th feature summary set, where hi represents the order of the feature summary results in the i-th feature summary set. The first perspective confidence score nh1={nh11,nh12,nh13,...,nh1h1} refers to the confidence score set of the feature summary results obtained by evaluating the confidence score of the first feature summary set, where nh11 is the first feature summary result in the first perspective confidence score set. The confidence scores for the results are as follows: nh12 represents the confidence score of the second feature summary result in the first-perspective confidence score, nh13 represents the confidence score of the third feature summary result in the first-perspective confidence score, and nh1h1 represents the confidence score of the h1th feature summary result in the first-perspective confidence score. The second-perspective confidence score nh2 = {nh21, nh22, nh23, ..., nh2h2} refers to the set of confidence scores for feature summary results obtained by evaluating the confidence scores of the second feature summary set, where nh21 represents the confidence score of the first feature summary result in the second-perspective confidence score, nh22 represents the confidence score of the second feature summary result in the second-perspective confidence score, and nh23 represents the confidence score of the third feature summary result in the second-perspective confidence score. The confidence level, nh2h2, is the confidence level of the h2th feature summary result in the second-perspective confidence level. The third-perspective confidence level nh3={nh31,nh32,nh33,...,nh3h3} refers to the set of confidence levels of feature summary results obtained by evaluating the confidence level of the third feature summary set. Here, nh31 is the confidence level of the first feature summary result in the third-perspective confidence level, nh32 is the confidence level of the second feature summary result in the third-perspective confidence level, nh33 is the confidence level of the third feature summary result in the third-perspective confidence level, and nh3h3 is the confidence level of the h3th feature summary result in the third-perspective confidence level. The first-perspective confidence level weight β1={β11, β12, β13,...β1h1} refers to the set of weight coefficients used to calculate the first confidence level. Here, β11 is the weight coefficient in the first-view confidence weight that measures the confidence level of the first feature summary result in the first-view confidence level; β12 is the weight coefficient in the first-view confidence weight that measures the confidence level of the second feature summary result in the first-view confidence level; β13 is the weight coefficient in the first-view confidence weight that measures the confidence level of the third feature summary result in the first-view confidence level; and β1h1 is the weight coefficient in the first-view confidence weight that measures the confidence level of the h1th feature summary result in the first-view confidence level. This embodiment does not limit the weight of the first-view confidence level; those skilled in the art can freely choose according to actual needs, as long as β1... The requirement of 1+β12+β13+,...,+β1h1=1 is sufficient. When h1=4, β11=0.25, β12=0.23, β13=0.26, β14=0.26. The second perspective confidence weight β2={β21, β22, β23,...,β2h2} refers to the set of weight coefficients used to calculate the second confidence. Among them, β21 is the weight coefficient that measures the confidence of the first feature summary result in the second perspective confidence, β22 is the weight coefficient that measures the confidence of the second feature summary result in the second perspective confidence, and β23 is the weight coefficient that measures the confidence of the third feature summary result in the second perspective confidence. The weighting coefficient for the confidence level, β2h2, is the weighting coefficient in the second-view confidence level weight that measures the confidence level of the h2th feature summary result in the second-view confidence level. This embodiment does not limit the weighting of the second-view confidence level; those skilled in the art can freely choose according to actual needs, as long as the requirement of β21+β22+β23+,...,+β2h2=1 is met. When h2=4, β21=0.25, β22=0.23, β23=0.26, β24=0.26 are set. The third-view confidence level weight β3={β31, β32, β33,...,β3h3} refers to the set of weighting coefficients calculated for the third confidence level, where β31 is the third-view confidence level weight. The weighting coefficients for the first feature summary result in the third-view confidence weighting are as follows: β32 is the weighting coefficient for the second feature summary result in the third-view confidence weighting; β33 is the weighting coefficient for the third feature summary result in the third-view confidence weighting; and β3h3 is the weighting coefficient for the h3th feature summary result in the third-view confidence weighting. This embodiment does not limit the weighting of the third-view confidence weights; those skilled in the art can freely choose according to actual needs, as long as the requirement of β31+β32+β33+,...,+β3h3=1 is met. When h3=4, β31 is set to 0.25, β32=0.23, β33=0.26, β34=0.26, the confidence weight βi={βi1, βi2, βi3,...,βihi} of the i-th viewpoint refers to the set of weight coefficients used to calculate the confidence of the i-th confidence. Here, βi1 is the weight coefficient in the confidence weight of the i-th viewpoint that measures the confidence of the first feature summary result, βi2 is the weight coefficient in the confidence weight of the i-th viewpoint that measures the confidence of the second feature summary result, βi3 is the weight coefficient in the confidence weight of the i-th viewpoint that measures the confidence of the third feature summary result, and βihi is the weight coefficient in the confidence weight of the i-th viewpoint that measures the confidence of the third feature summary result. The weighting coefficient of the confidence score of the hi-th feature summary result in the confidence score of the i-th viewpoint is given. This embodiment does not limit the weighting of the confidence score of the i-th viewpoint; those skilled in the art can freely choose according to actual needs, as long as the requirement of βi1+βi2+βi3+,...,+βihi=1 is met. When hi=4, βi1=0.25, βi2=0.23, βi3=0.26, βi4=0.26 are set. This embodiment does not limit the specific method of obtaining the maximum confidence score Bax; those skilled in the art can freely choose according to actual needs, such as assigning the first confidence score Ba1, the second confidence score Ba2, the third confidence score Ba3,...The confidence scores Bai of the i-th term are arranged in descending order to obtain the maximum confidence score. This embodiment does not limit the specific method of confidence assessment; those skilled in the art can freely choose according to actual needs, such as expert assessment. Expert assessment refers to the method by which experts with the ability to assess confidence scores conduct the assessment. This embodiment does not limit the specific implementation method of expert assessment; those skilled in the art can freely choose according to actual needs. For example, the feature summary set to be assessed is pushed to the expert interaction terminal via the cloud, and the expert's response is obtained during the expert interaction. The confidence score of the feature summary result input from the terminal is given by: nh1 being the first-view confidence score, nh2 being the second-view confidence score, nh3 being the third-view confidence score, and nhi being the i-th-view confidence score. The first-view confidence score weight is a coefficient measuring the importance of the first-view confidence score in the overall confidence score. The second-view confidence score weight is a coefficient measuring the importance of the second-view confidence score in the overall confidence score. The third-view confidence score weight is a coefficient measuring the importance of the third-view confidence score in the overall confidence score. The coefficients, where the confidence weight of the i-th viewpoint refers to the coefficient that measures the importance of the confidence of the i-th viewpoint in the overall confidence score, are not limited in this embodiment to the specific values ​​of the confidence weights β1 (first viewpoint), β2 (second viewpoint), β3 (third viewpoint), and βi (i-th viewpoint). Those skilled in the art can freely choose according to actual needs, as long as the requirement of β1 + β2 + β3 + ... + βi is met. For example, when i = 3, β1 = 0.4, β2 = 0.3, and β3 = 0.3 are set. The pre-set confidence score refers to the level of confidence achieved. The preset values ​​for judgment are not limited in this embodiment. Those skilled in the art can freely choose the preset confidence level according to actual needs. For example, this embodiment sets the preset confidence level Bax0=90. The "confidence level achievement status" refers to the degree of confidence achievement, which includes achieving and not achieving the standard. The summary review refers to the process of checking and updating the output process of each data block summary to output the result data block summary. This embodiment does not limit the specific method of summary review; those skilled in the art can freely choose according to actual needs, such as manual review.

[0124] Specifically, in step S3, by acquiring the confidence level and determining its compliance, a result data block summary is obtained to reduce data heterogeneity, unify the data representation, improve the feature extraction efficiency of multimodal data, focus on key information, and thus improve the efficiency of the robot's output working software system.

[0125] Specifically, in step S3, when anchoring the fused modal data block, the fused modal data block is input into the key scanning model to obtain the key entity.

[0126] Specifically, the key scanning model refers to a convolutional neural network model that takes fused modal data blocks as input data and key entities as output data. This embodiment does not limit the specific construction method of the key scanning model. Those skilled in the art can freely choose according to actual needs. For example, the convolutional neural network model can be trained by using historical fused modal data blocks and their corresponding key entities as training sets to obtain the key scanning model. The key entity refers to the semantic unit in the fused modal data block that can directly determine the semantics of the data content. For example, when the robot is applied in an educational scenario, the key entity is the student's highlighted notes from the fused modal data block of the student's electronic notes text.

[0127] Specifically, in step S3, key entities are acquired to retain key information, adapt to different task requirements, and efficiently process multimodal data, thereby improving the efficiency of the robot's output working software system.

[0128] Specifically, in step S3, when adjusting the result data block summary based on key entities, the entity proportion coefficient Y is obtained based on the key entities, and the entity proportion coefficient Y is compared with the preset entity proportion coefficient Y0. Based on the comparison result, the compliance status of the entity proportion coefficient is judged, and the preset reliability Bax0 is adjusted based on the judgment result, wherein:

[0129] When Y≥Y0, the entity proportion coefficient is deemed to have met the standard, and no adjustment is made to the pre-set confidence level Bax0.

[0130] When Y < Y0, the entity proportion coefficient is deemed unqualified. The preset reliability Bax0 is adjusted based on the adjustment coefficient jg, where jg = 1.57 - 0.23 × e -0.38×(Y0-Y) Where e is the base of the natural logarithm, the adjusted preset confidence level Bax0` is obtained, Bax0` is set to Bax0×jg, the preset confidence level Bax0 is replaced with the adjusted preset confidence level Bax0`, and the maximum confidence level Bax is re-compared with the adjusted preset confidence level Bax0`.

[0131] Specifically, the entity proportion coefficient refers to a value used to measure the proportion of key entities in the fused modal data block. This embodiment does not limit the specific method of obtaining the entity proportion coefficient. Those skilled in the art can freely choose according to actual needs. For example, if the total number of text data in the fused modal data block is set to wb and the total number of key entities in the text data is set to wg, then the entity proportion coefficient Y = wg / wb. The preset entity proportion coefficient refers to a preset value for judging the compliance status of the entity proportion coefficient. This embodiment does not limit the preset entity proportion coefficient. Those skilled in the art can freely choose according to actual needs. For example, in this embodiment, the preset entity proportion coefficient Y0 = 0.85 is set. The compliance status of the entity proportion coefficient refers to the degree of compliance of the entity proportion coefficient obtained by comparing the entity proportion coefficient with the preset entity proportion coefficient. The compliance status of the entity proportion coefficient includes compliance and non-compliance.

[0132] Specifically, in step S3, by judging whether the entity proportion coefficient meets the standard, in the early stage when the entity proportion coefficient does not meet the standard, the preset confidence is rapidly increased according to the result adjustment coefficient to adjust the result in a timely manner and maintain system stability. In the later stage, the change in the entity proportion coefficient tends to level off, and the result adjustment coefficient value stabilizes at 0.57 after rising, so as to match the overall trend of result adjustment, so as to adjust the result data block summary according to necessary and key data, thereby improving the accuracy and stability of the output working software system.

[0133] Specifically, in step S3, when updating the results based on the destructive scenario, an initial destructive scenario is set up to obtain an initial destructive scenario. The fused modal data block is compared with the initial destructive scenario. Based on the comparison result, the existence of the fused modal data block in the initial destructive scenario is determined, and the key entities are updated based on the determination result, wherein:

[0134] When the existence of the fused modal data block is not found in the initial destructive scenario, no entity update is performed on the key entity;

[0135] When the existence of the fused modal data block is confirmed in the initial destructive scenario, the key entities are updated.

[0136] Specifically, when updating key entities, destructive text is removed to obtain a standard fusion modality data block, which is then re-inputted into the key scanning model to obtain the updated key entity.

[0137] Specifically, the destructive scenario refers to a preset text database for identifying destructive text in the fused modal data block. The destructive text refers to unconventional text instances that may cause analysis errors. The initial setting refers to the process of setting preset scenario data for detecting destructive text within the destructive scenario. This embodiment does not limit the initial setting; those skilled in the art can freely choose according to actual needs, such as setting adversarial interference. Adversarial interference refers to deliberately designed grammatical errors and unconventional symbol insertions, such as "true values ​​are distorted" or "the above content does not conform." The existence of the fused modal data block in the initial destructive scenario refers to whether the text data in the fused modal data block obtained by comparing the fused modal data block with the text data in the initial destructive scenario overlaps. This embodiment does not determine whether the text data in the fused modal data block overlaps with the text data in the initial destructive scenario. The method is limited, such as inputting the text data in the fused modality data block and the text data in the initial destructive scenario into the Paython tool for bag-of-words model comparison to obtain the similarity between the text data in the fused modality data block and the text data in the initial destructive scenario. When the similarity between the text data in the fused modality data block and the text data in the initial destructive scenario is greater than 90%, it is determined to be overlapping. The existence of the fused modality data block in the initial destructive scenario includes the absence of the fused modality data block in the initial destructive scenario and the presence of the fused modality data block in the initial destructive scenario. The destructive text refers to text data in the fused modality data block that has been maliciously tampered with, forged, or contains harmful information. This embodiment does not limit the specific method of removing destructive text. Those skilled in the art can freely choose according to actual needs, such as removing it through a text cleaning library. The text cleaning library refers to a Python tool library that integrates multiple text cleaning rules.

[0138] Specifically, in step S3, destructive text in the fused modal data block is removed through destructive scenarios, so that non-standard text that causes analysis errors can be detected by preset rules and the result data summary can be updated in a timely manner, thereby protecting the system and improving the security and rationality of the output software system.

[0139] Specifically, in step S4, when adjusting the bias during the process of data stratification based on the bias coefficient, the bias is adjusted according to the probability of the fused modal data block type. ={ , , ,..., } and total number of categories Bias coefficient Perform calculations and set Obtain the bias coefficient , bias coefficient The bias coefficient is compared with the preset bias coefficient Pmg0. Based on the comparison results, the compliance of the bias coefficient is judged, and the bias of the number of viewpoints n0 is adjusted according to the judgment results.

[0140] when When the value is ≥Pmg0, step S4 determines that the bias coefficient meets the standard and does not adjust the number of viewpoints n0 for bias.

[0141] when When <Pmg0, step S4 determines that the bias coefficient does not meet the standard, and adjusts the bias of the number of viewpoints n0 to obtain the adjusted number of viewpoints n0`. Set n0` = n0 + 1, and re-layer the data and adjust the bias according to the adjusted number of viewpoints n0` and the newly added analysis viewpoints until step S4 determines that the bias coefficient meets the standard.

[0142] Specifically, the fusion modality data block type probability refers to the set of proportions of the occurrence of a single text data type in the fusion modality data block relative to the total number of categories. The total number of categories refers to the total number of text data types obtained after classifying the text data in the fusion modality data block. Here, p1 is the proportion of the first sample type in the fusion modality data block type probability relative to the total number of categories, p2 is the proportion of the second sample type in the fusion modality data block type probability relative to the total number of categories, p3 is the proportion of the third sample type in the fusion modality data block type probability relative to the total number of categories, and pj is the proportion of the j-th sample type in the fusion modality data block type probability relative to the total number of categories. This embodiment does not limit the specific method of obtaining the text data in the fusion modality data block for classification; those skilled in the art can determine the appropriate method based on actual conditions. The requirements can be freely chosen, such as classifying text data according to its source, or treating the text data obtained after keyframe segmentation and modal conversion of the target video as a single text data type. The bias coefficient is a coefficient used to measure the uniformity of text data distribution in the fused modal data block, calculated based on the probability of the fused modal data block type and the total number of categories. The preset bias coefficient is a preset value for judging the compliance status of the bias coefficient. This embodiment does not limit the specific value setting of the preset bias coefficient. Those skilled in the art can freely choose according to actual needs. For example, this embodiment sets the preset bias coefficient P0=2.5. The compliance status of the bias coefficient refers to the degree of compliance of the bias coefficient obtained by judging the bias coefficient and the preset bias coefficient. The compliance status of the bias coefficient includes compliance and non-compliance.

[0143] Specifically, in step S4, the compliance of the bias coefficient is judged. When the bias coefficient does not meet the standard, the number of viewpoints is increased one by one to increase the accuracy of data stratification. This is to avoid the unevenness of the data affecting the effective data stratification, thereby improving the accuracy and efficiency of the output software system.

[0144] Specifically, in step S4, when updating the bias according to the strict scenario of the entity, the key entity scenario is compared with the strict scenario library. Based on the comparison result, the existence of the key entity scenario in the strict scenario library is determined, and the entity strict scenario is output based on the determination result. Furthermore, the preset bias coefficient Pmg0 is updated based on the determination result, wherein:

[0145] When step S4 determines that the existence of the key entity scene in the strict scene library is non-existent, the entity strict scene is not output, and the preset bias coefficient P0 is not updated.

[0146] When step S4 determines that the existence of the key entity scene in the strict scene library is true, the entity strict scene is output and the preset bias coefficient Pmg0 is updated.

[0147] Specifically, when updating the preset bias coefficient Pmg0, the preset bias coefficient Pmg0 is updated according to the bias update coefficient pg to obtain the updated preset bias coefficient Pmg0`. Pmg0` is set to Pmg0×pg. The preset bias coefficient Pmg0 is replaced with the updated preset bias coefficient Pmg0`, and the bias coefficient Pmg is re-compared with the updated preset bias coefficient Pmg0`.

[0148] Specifically, the key entity scenario refers to a specific scenario surrounding a key entity, such as a contract signing scenario. This embodiment does not limit the specific method of obtaining the key entity scenario; those skilled in the art can freely choose according to actual needs, such as analyzing and judging based on a large language model and outputting the key entity scenario result. The strict scenario library refers to a set of preset scenario text data for judging strict scenarios of entities. This embodiment does not limit the construction method of the strict scenario library; those skilled in the art can freely choose according to actual needs, such as expert setting. Expert setting refers to the process of constructing the strict scenario library by experts with the ability to set preset scenario text data in the strict scenario library. This embodiment does not include expert setting. The specific method for obtaining the strict scene library is limited, and those skilled in the art can freely choose according to actual needs. For example, it can be obtained through the cloud from the strict scene library input by experts in the expert setting terminal. The existence of the key entity scene in the strict scene library refers to whether there is text data with the same key entity scene in the strict scene library. The existence of the key entity scene in the strict scene library includes non-existence and existence. The bias update coefficient refers to the preset coefficient value for updating the preset bias coefficient. This embodiment does not limit the preset bias coefficient, and those skilled in the art can freely choose according to actual needs, as long as the need to increase the preset bias coefficient is met. For example, in this embodiment, the bias update coefficient pg=1.35 is set.

[0149] Specifically, in step S4, the existence of key entity scenarios in the strict scenario library is judged. When a key entity scenario exists in the strict scenario library, the entity strict scenario is output, and the preset bias coefficient is increased to avoid the logical defects of the key entity scenario affecting the compliance of the bias coefficient, thereby improving the accuracy of the output working software system.

[0150] Specifically, in step S4, a memory map is constructed based on the result data block summary using a memory map construction method, the memory map construction method including:

[0151] Step S41: Construct the knowledge base to obtain the basic knowledge base;

[0152] Step S42: Compare the result data block summary with the basic knowledge base to obtain a preliminary memory map;

[0153] Step S42: Based on the repair coefficient xb, perform contradiction repair on the preliminary memory map to obtain the memory map.

[0154] Specifically, the knowledge base refers to a database built upon the basic framework of a memory graph, using associated attribute paths as its content. The associated attribute paths refer to the connection paths between text data, such as Einstein-occupation-physicist. This embodiment does not limit the specific construction method of the knowledge base; those skilled in the art can freely choose according to actual needs, such as having experts set the associated attributes in the knowledge base. The entity comparison refers to comparing the result data block summary with the associated attribute paths in the basic knowledge base. When the text proportion of path words in the result data block summary and the associated attribute paths in the basic knowledge base reaches a preset proportion zb, such as zb=10%, it is identified as the result data block summary of the corresponding attribute, and the result data block summaries of corresponding attributes on the same associated attribute path are concatenated. The path vocabulary of the associated attribute path refers to a single associated word in the associated attribute path, such as Einstein-occupation-physicist, Einstein, occupation and physicist. The repair coefficient refers to a value set to represent the repair strength for contradictory repair of the preliminary memory map. This embodiment does not limit the repair coefficient, and those skilled in the art can freely choose it according to actual needs. For example, in this embodiment, the repair coefficient xb is set to 1.27. The contradictory repair refers to the process of correcting the associated attributes in the preliminary memory map to improve the output accuracy of the memory map. For example, the preset proportion zb is repaired according to the repair coefficient xb to obtain the repaired preset proportion zb`. zb` is set to zb×xb. The preset proportion zb is replaced with the repaired preset proportion zb`, and the entity comparison is re-performed according to the repaired preset proportion zb`.

[0155] Specifically, in step S4, by constructing the memory map, the result data block summary is associated and integrated to enhance the robot's semantic understanding and reasoning ability, thereby improving the efficiency of the robot's software system.

[0156] Specifically, in step S5, the working software system is output based on the memory map and multimodal data.

[0157] It is understood that this embodiment does not limit the specific method of outputting the working software system based on the memory map and multimodal data. Those skilled in the art can freely choose according to actual needs, such as inputting the memory map and multimodal data into a knowledge management tool to generate the working software system, or inputting the memory map and multimodal data into Python+Flask to obtain the working software system.

[0158] Specifically, in step S5, the target customer's needs are realistically transformed by outputting the working software system, thereby obtaining a usable working software system, which improves the integrity of the robot's usage process and the output efficiency of the working software system.

[0159] Specifically, in step S5, when correcting the output process of the working software system according to the real-time software generation requirements, the real-time software generation requirement ts is compared with the previous real-time software generation requirement ts0. Based on the comparison result, the update status of the multimodal data is judged, and based on the judgment result, the multimodal data is corrected. Wherein:

[0160] If the update status of the multimodal data is determined to be no update, no working correction is performed on the multimodal data;

[0161] When the update status of the multimodal data is determined to be updated, the multimodal data is corrected to obtain corrected multimodal data. Steps S2 to S5 are repeated based on the corrected multimodal data to obtain the corrected working software system. The corrected working software system is then output as the working software system.

[0162] Specifically, the "previous real-time software generation requirement" refers to the real-time software generation requirement output by the working software system in the previous moment. This embodiment does not limit the specific method of obtaining the previous real-time software generation requirement; those skilled in the art can freely choose according to actual needs. For example, the time of the previous real-time software generation requirement can be obtained based on the time step nfd. This embodiment does not limit the setting of the time step nfd; for example, if nfd = 2 seconds and the current time of generating the real-time software generation requirement is set to fm, then the time of the previous real-time software generation requirement is fm - nfd. This embodiment does not limit the specific method of judging the update status of multimodal data; for example, the real-time software generation requirement can be compared with the previous real-time software generation requirement. The Diffchecker is used to obtain preliminary update results. These results are then input into Python to obtain the update difference coefficient qy. Updates with qy ≤ 0.2 are considered as no update, and updates with qy > 0.2 are considered as updates. The update status of the multimodal data refers to whether the multimodal data is updated due to whether the target user has a new real-time software generation requirement. The update status of the multimodal data includes no update and updates. This embodiment does not limit the specific method of working correction of the multimodal data. Those skilled in the art can freely choose according to actual needs, such as adding the real-time software generation requirement to the previous real-time software generation requirement and repeating steps S1 to S5 to obtain the corrected multimodal data.

[0163] Specifically, in step S5, by judging the update status of multimodal data, when there is an update to the multimodal data, the multimodal data is promptly corrected to facilitate timely updates to the target user's software generation requirements, thereby improving the adaptability and efficiency of the robot generation software system.

[0164] Specifically, in step S5, when performing difference optimization on the memory map and data stratification based on the difference coefficient, if the update status of the multimodal data is "updated", the difference coefficient CY is calculated based on the time step nfd, the update difference coefficient qy, the working software system similarity rf, the time step weight nm1, the update difference coefficient weight nm2, and the working software system similarity weight nm3. The result is set as CY = nfd × nm1 + qy × nm2 + rf × nm3. The difference coefficient CY is then compared with the first preset difference coefficient CY1 and the second preset difference coefficient CY2. Based on the comparison result, the difference in the difference coefficient is judged, and based on the judgment result, the repair coefficient xb and the number of viewpoints n0 are optimized.

[0165] When CY≤CY1, the difference in the difference coefficient is determined to be a low degree of deviation, and no difference optimization is performed on the repair coefficient xb and the number of viewpoints n0.

[0166] When CY1 < CY ≤ CY2, the difference in the difference coefficient is determined to be a moderate deviation. The repair coefficient xb is then optimized based on the first difference coefficient dy, with dy = 1.28 - 0.22 × e. -0.48×(CY-CY1) Where e is the base of the natural logarithm, the optimized repair coefficient xb` is obtained, xb` is set to xb×dy, the repair coefficient xb is replaced with the optimized repair coefficient xb`, and the contradiction is repaired again on the preliminary memory map according to the optimized repair coefficient xb`;

[0167] When CY > CY2, the difference in the difference coefficient is determined to be a height deviation. Difference optimization is then performed on the number of viewing angles n0, based on the second difference coefficient df. df is set to 1.61 - 0.28 × e. -0.66×(CY-CY2) Where e is the base of the natural logarithm, the optimized number of viewpoints n0`` is obtained, and n0`` is set to n0×df. The value of n0`` is rounded to a positive integer. The number of viewpoints n0 is replaced with the optimized number of viewpoints n0``, and the fused modal data block is redefined based on the optimized number of viewpoints n0`` and the newly added analysis viewpoints.

[0168] Specifically, the working software system similarity refers to the degree of similarity between the currently generated working software system and the working software system at the previous moment. This embodiment does not limit the specific method of determining the working software system similarity. For example, inputting the currently generated working software system and the working software system at the previous moment into Diffchecker yields a preliminary similarity result, and inputting the preliminary similarity result into Python yields the working software system similarity. The time step weight is a coefficient that measures the importance of the time step in the difference coefficient. The update difference coefficient weight is a coefficient that measures the importance of the update difference coefficient in the difference coefficient. The working software system similarity weight is a coefficient that measures the importance of the working software system similarity in the difference coefficient. This embodiment does not limit the specific numerical settings of the time step weight nm1, the update difference coefficient weight nm2, and the working software system similarity weight nm3. For example, in this embodiment, nm1=0.2, nm2=0.3, and nm3=0.3 are set. The first preset difference coefficient is a preset value used to judge the difference in the difference coefficient. The lower limit, the second preset difference coefficient refers to the upper limit of the preset value for judging the difference of the difference coefficient. This embodiment does not limit the first preset difference coefficient and the second preset difference coefficient. Those skilled in the art can freely choose according to actual needs. For example, in this embodiment, the first preset difference coefficient CY1=0.23 and the second preset difference coefficient CY2=0.31 are set. The difference of the real-time software generated requirements refers to the degree of deviation of the target user's requirements judged based on the time step, the update difference coefficient and the similarity of the working software system. The difference of the difference coefficient includes low deviation, medium deviation and high deviation. The new analysis perspective refers to the newly added analysis perspective that is different from the analysis perspective at the previous moment after bias adjustment. This embodiment does not limit the specific acquisition method of the new analysis perspective. For example, it can be obtained through expert setting. The expert setting refers to an expert with the ability to set the new analysis perspective. This embodiment does not limit the specific acquisition method of expert setting. For example, it can be obtained through the cloud from the new analysis perspective entered by the expert in the interactive terminal window.

[0169] Specifically, in step S5, the difference in the difference coefficient is judged. When the difference in the difference coefficient is in the middle stage, a first difference coefficient is set to optimize the repair coefficient. When the difference in the difference coefficient reaches the initial stage of the middle stage, the repair coefficient needs to be increased rapidly to adjust the memory map in a timely manner. When the difference in the difference coefficient reaches the later stage of the middle stage, the rate of change tends to be stable, and a first difference coefficient is set to match the trend of change, so as to reasonably avoid the impact of the difference in the real-time demand vector on the construction of the memory map. When the difference in the difference coefficient is large, a second difference coefficient is set to optimize the number of views. When the difference in the difference coefficient reaches the initial stage of the large stage, the number of views needs to be increased rapidly to re-layer the data in a timely manner. When the difference in the difference coefficient reaches the later stage of the large stage, the trend of change tends to be stable, and a second difference coefficient is set to match this trend, so as to reasonably avoid the large difference in the difference coefficient from affecting the accuracy of data layering, thereby improving the accuracy and efficiency of the robot generation software system.

[0170] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A software system development method based on a multimodal AI large model, characterized in that, The method includes: Step S1: Acquire multimodal data; Step S2: Preprocess the multimodal data according to the preprocessing method to obtain processed multimodal data, and also obtain the fused modal data block based on the processed multimodal data; Step S3: Perform data layering on the fused modal data block to obtain a result data block summary; perform anchor scanning on the fused modal data block to obtain key entities; adjust the result data block summary based on the key entities; and update the result adjustment based on the destructive scenario. Step S4 involves adjusting the bias of the data stratification process based on the bias coefficient, updating the bias of the bias adjustment process based on the strict scenario of the entity, and constructing the memory graph based on the result data block summary using the memory graph construction method to obtain the memory graph. Step S5: Output the working software system based on the memory map and multimodal data, and make working corrections to the output process of the working software system according to the real-time software generation requirements, and optimize the memory map and data layering based on the difference coefficient. The destructive scenario refers to a pre-defined text database for identifying destructive text in the fused modal data block, and the destructive text refers to unconventional text instances that may cause analysis errors. The key entity scenario is compared with the strict scenario library. Based on the comparison results, the existence of the key entity scenario in the strict scenario library is judged, and the entity strict scenario is output based on the judgment results. The strict scenario library refers to a set of preset scenario text data for judging entity strict scenarios.

2. The software system development method based on a multimodal AI large model according to claim 1, characterized in that, The multimodal data is preprocessed according to a preprocessing method, the preprocessing method including: Step A01: Perform text cleaning on the target text in the multimodal data to obtain the processed target text; Step A02: Extract keyframes from the target video in the multimodal data to obtain target keyframes, extract motion features from the target keyframes to obtain the processed target video; Step A03: Denoise the target speech in the multimodal data to obtain the denoised target speech, and extract speech features from the denoised target speech to obtain the processed target speech. Step A04: Unify the format of the target image in the multimodal data to obtain the unified target image, and normalize the unified target image to obtain the processed target image; Step A05: The processed target text, processed target video, processed target audio, processed target image, and real-time software-generated requirements are taken as processed multimodal data.

3. The software system development method based on a multimodal AI large model according to claim 2, characterized in that, In step S2, when acquiring the fused modal data block based on the processed multimodal data, the processed multimodal data is divided into data blocks using a data segmentation method, which includes: Step B01: Perform text conversion on the processed target speech to obtain the second target text; perform length block division on the second target text and the processed target text to obtain text modality blocks. Step B02: Perform keyframe segmentation on the processed target video to obtain video modal segments; Step B03: Divide the processed target image into grid blocks to obtain image modal blocks; Step B04: The text modal blocks, video modal blocks, and image modal blocks are used as multimodal data blocks.

4. The software system development method based on a multimodal AI large model according to claim 3, characterized in that, In step S2, when acquiring the fused modal data block based on the processed multimodal data, the multimodal data block is modally converted according to a modal conversion method, the modal conversion method including: Step S21: Input the video modal blocks into the target detection model to obtain video text blocks, and add timestamps to the video text blocks to obtain target video text blocks; Step S22: Input the image modal blocks into the visual language model to obtain the target image text blocks; Step S23: The text modal blocks, target video text blocks, and target image text blocks are used as fused modal data blocks.

5. The software system development method based on a multimodal AI large model according to claim 4, characterized in that, In step S3, when performing data layering on the fused modal data block, the fused modal data block is layered according to a data layering method, which includes: Step S31: Set the number of viewpoints n0. Initialize the definition of the fused modal data blocks according to the number of viewpoints n0 and the analysis viewpoints to obtain the data block summary nf={nf1,nf2,nf3,...,nfi}, where nf1 is the first feature summary set in each data block summary, set nf1={nf11,nf12,nf13,...,nf1h1}, nf2 is the second feature summary set in each data block summary, set nf2={nf21,nf22,nf23,...,nf2h2}, nf3 is the third feature summary set in each data block summary, set nf3={nf31,nf32,nf33,...,nf3h3},...,nfi is the i-th feature summary set in each data block summary, set nfi={nfi1,nfi2,nfi3,...,nfihi}; Step S32: Obtain the first viewpoint confidence score nh1={nh11,nh12,nh13,...,nh1h1} based on the first feature summary set nf1={nf11,nh12,nh13,...,nh1h1}; The confidence scores nh2={nh21,nh22,nh23,...,nh2h2} of the second perspective are obtained based on the second feature summary set nf2={nf21,nh22,nh23,...,nh2h2}. The confidence scores of the third perspective nh3={nh31,nh32,nh33,...,nh3h3} are obtained based on the third feature summary set nf3={nf31,nh32,nh33,...,nh3h3}. …… The confidence scores nhi = {nhi1, nhi2, nhi3, ..., nhih3} of the i-th viewpoint are obtained based on the i-th feature summary set nfi = {nfi1, nhi2, nhi3, ..., nhih3}. Step S33: Calculate the first confidence level Ba1 based on the first perspective confidence level nh1={nh11,nh12,nh13,...,nh1h1}, the total number of first feature summary results n1, and the first perspective confidence level weight β1={β11, β12, β13,...,β1h1}, and set... The first confidence level Ba1 is obtained; The confidence level Ba2 is calculated based on the second-view confidence level nh2={nh21,nh22,nh23,...,nh2h2}, the total number of second feature summary results n2, and the second-view confidence level weight β2={β21, β22, β23,...,β2h2}. The second confidence level, Ba2, is obtained. The confidence level Ba3 is calculated based on the third-view confidence level nh3={nh31,nh32,nh33,...,nh3h3}, the total number of third-feature summary results n3, and the third-view confidence weight β3={β31,β32,β33,...,β3h3}. The third confidence level, Ba3, is obtained. …… The confidence score Bai is calculated based on the confidence score nhi={nhi1,nhi2,nhi3,...,nhih3} of the i-th perspective, the total number of i-th feature summary results ni, and the confidence score weight βi={βi1,βi2,βi3,...,βihi} of the i-th perspective. We obtain the confidence level Bai of the i-th term; Step S34: Obtain the maximum confidence Bax based on the first confidence Ba1, the second confidence Ba2, the third confidence Ba3, ..., the i-th confidence Bai; Step S35: Compare the maximum confidence level Bax with the preset confidence level Bax0. Based on the comparison result, determine whether the maximum confidence level meets the standard, and output the result data block summary based on the determination result. When Bax≥Bax0, the maximum confidence level is considered met, and the feature summary set corresponding to the maximum confidence level is output as the result data block summary. When Bax < Bax0, the maximum confidence level is deemed unqualified. The maximum confidence level is then reviewed to obtain the reviewed maximum confidence level. The manually input feature summary set is then output as the result data block summary. In step S3, when performing anchor scanning on the fused modal data block, the fused modal data block is input into the key scanning model to obtain the key entity.

6. The software system development method based on a multimodal AI large model according to claim 5, characterized in that, In step S3, when adjusting the result data block summary based on key entities, the entity proportion coefficient Y is obtained based on the key entities. The entity proportion coefficient Y is compared with the preset entity proportion coefficient Y0. Based on the comparison result, the compliance status of the entity proportion coefficient is judged, and the preset reliability Bax0 is adjusted based on the judgment result. Wherein: When Y≥Y0, the entity proportion coefficient is deemed to have met the standard, and no adjustment is made to the pre-set confidence level Bax0. When Y < Y0, the entity proportion coefficient is deemed unqualified. The preset reliability Bax0 is adjusted based on the adjustment coefficient jg, where jg = 1.57 - 0.23 × e -0.38×(Y0-Y) Where e is the base of the natural logarithm, the adjusted preset confidence level Bax0` is obtained, Bax0` is set to Bax0×jg, the preset confidence level Bax0 is replaced with the adjusted preset confidence level Bax0`, and the maximum confidence level Bax is recompared with the adjusted preset confidence level Bax0`. The entity proportion coefficient refers to a value used to measure the proportion of key entities in the fused modal data block; In step S3, when updating the results based on the destructive scenario, an initial destructive scenario is set up to obtain an initial destructive scenario. The fused modal data block is compared with the initial destructive scenario. Based on the comparison result, the existence of the fused modal data block in the initial destructive scenario is determined, and the key entities are updated based on the determination result, wherein: When the existence of the fused modal data block is not found in the initial destructive scenario, no entity update is performed on the key entity; When the existence of the fused modal data block is confirmed in the initial destructive scenario, the key entities are updated. Specifically, when updating key entities, destructive text is removed to obtain a standard fusion modality data block, which is then re-inputted into the key scanning model to obtain the updated key entity.

7. The software system development method based on a multimodal AI large model according to claim 6, characterized in that, In step S4, when bias adjustment is performed on the data stratification process based on the bias coefficient, the probability of the fused modal data block type is considered. ={ , , ,..., } and total number of categories Bias coefficient Perform calculations and set Obtain the bias coefficient , bias coefficient The bias coefficient is compared with the preset bias coefficient Pmg0. Based on the comparison results, the compliance of the bias coefficient is judged, and the bias of the number of viewpoints n0 is adjusted according to the judgment results. when When the value is ≥Pmg0, step S4 determines that the bias coefficient meets the standard and does not adjust the number of viewpoints n0 for bias. when When <Pmg0, step S4 determines that the bias coefficient does not meet the standard, and adjusts the bias of the number of viewpoints n0 to obtain the adjusted number of viewpoints n0`. Set n0` = n0 + 1, and re-layer the data and adjust the bias according to the adjusted number of viewpoints n0` and the newly added analysis viewpoints until step S4 determines that the bias coefficient meets the standard.

8. The software system development method based on a multimodal AI large model according to claim 7, characterized in that, In step S4, during the bias update process based on the strict scenario of the entity, the key entity scenario is compared with the strict scenario library. Based on the comparison result, the existence of the key entity scenario in the strict scenario library is determined, and the entity strict scenario is output based on the determination result. Furthermore, the preset bias coefficient Pmg0 is updated based on the determination result, wherein: When step S4 determines that the existence of the key entity scene in the strict scene library is non-existent, the entity strict scene is not output, and the preset bias coefficient P0 is not updated. When step S4 determines that the existence of the key entity scene in the strict scene library is true, the entity strict scene is output and the preset bias coefficient Pmg0 is updated. Specifically, when updating the preset bias coefficient Pmg0, the preset bias coefficient Pmg0 is updated according to the bias update coefficient pg to obtain the updated preset bias coefficient Pmg0`. Pmg0` is set to Pmg0×pg. The preset bias coefficient Pmg0 is replaced with the updated preset bias coefficient Pmg0`, and the bias coefficient Pmg is re-compared with the updated preset bias coefficient Pmg0`.

9. The software system development method based on a multimodal AI large model according to claim 8, characterized in that, In step S4, a memory map is constructed based on the result data block summary using a memory map construction method, the memory map construction method including: Step S41: Construct the knowledge base to obtain the basic knowledge base; Step S42: Compare the result data block summary with the basic knowledge base to obtain a preliminary memory map; Step S42: Based on the repair coefficient xb, perform contradiction repair on the preliminary memory map to obtain the memory map.

10. The software system development method based on a multimodal AI large model according to claim 9, characterized in that, In step S5, the working software system outputs data based on the memory map and multimodal data; In step S5, when correcting the output process of the working software system according to the real-time software generation requirements, the real-time software generation requirement ts is compared with the previous real-time software generation requirement ts0. Based on the comparison result, the update status of the multimodal data is judged, and the multimodal data is corrected based on the judgment result. Specifically: If the update status of the multimodal data is determined to be no update, no working correction is performed on the multimodal data; When the update status of the multimodal data is determined to be updated, the multimodal data is corrected to obtain corrected multimodal data. Steps S2 to S5 are repeated based on the corrected multimodal data to obtain the corrected working software system. The corrected working software system is then output as the working software system. In step S5, when performing difference optimization on the memory map and data stratification based on the difference coefficient, if the update status of the multimodal data is "updated", the difference coefficient CY is calculated based on the time step nfd, the update difference coefficient qy, the working software system similarity rf, the time step weight nm1, the update difference coefficient weight nm2, and the working software system similarity weight nm3. The result is set as CY = nfd × nm1 + qy × nm2 + rf × nm3. The difference coefficient CY is then compared with the first preset difference coefficient CY1 and the second preset difference coefficient CY2. Based on the comparison result, the difference in the difference coefficient is judged, and based on the judgment result, the repair coefficient xb and the number of viewpoints n0 are optimized. When CY≤CY1, the difference in the difference coefficient is determined to be a low degree of deviation, and no difference optimization is performed on the repair coefficient xb and the number of viewpoints n0. When CY1 < CY ≤ CY2, the difference in the difference coefficient is determined to be a moderate deviation. The repair coefficient xb is then optimized based on the first difference coefficient dy, with dy = 1.28 - 0.22 × e. -0.48×(CY-CY1) Where e is the base of the natural logarithm, the optimized repair coefficient xb` is obtained, xb` is set to xb×dy, the repair coefficient xb is replaced with the optimized repair coefficient xb`, and the contradiction is repaired again on the preliminary memory map according to the optimized repair coefficient xb`; When CY > CY2, the difference in the difference coefficient is determined to be a height deviation. Difference optimization is then performed on the number of viewing angles n0, based on the second difference coefficient df. df is set to 1.61 - 0.28 × e. -0.66×(CY-CY2) Where e is the base of the natural logarithm, the optimized number of viewpoints n0`` is obtained, and n0`` is set to n0×df. The value of n0`` is rounded to a positive integer. The number of viewpoints n0 is replaced with the optimized number of viewpoints n0``, and the fused modal data block is redefined based on the optimized number of viewpoints n0`` and the newly added analysis viewpoints.

Images