Business implementation code generation method and device, storage medium and electronic equipment

Abstract

The invention discloses a service implementation code generation method and device, a storage medium and electronic equipment, and relates to the field of artificial intelligence, and the method comprises the steps: receiving a code generation request initiated by a target account, the code generation request being used for requesting to generate a service implementation code for realizing a service function indicated by target service data, the data modality of the target business data comprises at least one of the following modes: an image modality, an audio modality and a text modality; an initial service implementation code of the target service data is generated through a target code generation model, the target code generation model is obtained by training the initial code generation model through multi-mode service data marked with code samples, and the multi-mode service data comprises service image data, service audio data and service text data; adjusting the initial business implementation code; and sending the target service implementation code to the target account. Through the method and the device, the problem of relatively low generation efficiency of business implementation codes in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a method, device, storage medium and electronic device for generating a business implementation code. Background Art

[0002] With the continuous development of the field of artificial intelligence, the function of using code generation models to generate the required business code has brought great convenience to people's work and life. However, the generation effect of the current business code is very limited to the single-modal data provided by the user of the code generation model, which makes the use of the code generation model not very effective.

[0003] Currently, no effective solution has been proposed to the problem of low efficiency in generating business implementation codes in related technologies. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, storage medium and electronic device for generating a business implementation code, so as to solve the problem of low efficiency in generating business implementation codes in related technologies.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for generating a business implementation code is provided. The method comprises: receiving a code generation request initiated by a target account, wherein the code generation request is used to request the generation of a business implementation code for implementing a business function indicated by target business data, and the data modality of the target business data comprises at least one of the following: image modality, audio modality and text modality; responding to the code generation request, generating an initial business implementation code of the target business data through a target code generation model, wherein the target code generation model is obtained by training an initial code generation model using multimodal business data with code samples annotated, and the multimodal business data comprises: business image data, business audio data and business text data; adjusting the initial business implementation code to obtain a target business implementation code; and sending the target business implementation code to the target account.

[0006] As an optional implementation mode, the generating the initial business implementation code of the target business data through the target code generation model includes: extracting code condition information from the code generation request, wherein the code condition information is used to indicate the code parameter configuration and code language proposed by the target account for the generated code; inputting the code condition information and the target business data into the target code generation model; and obtaining the initial business implementation code output by the target code generation model that meets the code condition information.

[0007] As an optional implementation, before inputting the code condition information and the target business data into the target code generation model, the method also includes: using the business code and business data of the implemented business to construct multimodal business data annotated with code samples to obtain a training data set; and using the training data set to train the initial code generation model to obtain the target code generation model.

[0008] As an optional implementation manner, the multimodal business data annotated with code samples is constructed using the business code and business data of the implemented business to obtain a training data set, including: collecting the business code and business data of the implemented business; extracting business data of multiple data modes from the business data, and extracting the code samples corresponding to each business data from the business code, to obtain a first business code corresponding to the business image data in the image mode, a second business code corresponding to the business audio data in the audio mode, and a third business code corresponding to the business text data in the text mode, wherein the multiple data modalities include: the image mode, the audio mode and the text mode; annotating the first business code to the business image data, annotating the second business code to the business audio data, and annotating the third business code to the business text data to obtain the training data set.

[0009] As an optional implementation manner, the business data of multiple data modalities are extracted from the business data, and the code samples corresponding to each business data are extracted from the business code to obtain a first business code corresponding to the business image data in the image mode, a second business code corresponding to the business audio data in the audio mode, and a third business code corresponding to the business text data in the text mode, including: extracting interface data from the business data, and extracting a business implementation code of the interface data from the business code to obtain a first business code corresponding to the business image data in the image mode; extracting voice description data from the business data, and extracting a business implementation code of the voice description data from the business code to obtain a second business code corresponding to the business audio data in the audio mode; extracting text description data from the business data, and extracting a business implementation code of the text description data from the business code to obtain a third business code corresponding to the business text data in the text mode.

[0010] As an optional implementation manner, the step of marking the first business code to the business image data, marking the second business code to the business audio data, and marking the third business code to the business text data to obtain the training data set includes: extracting interface layout features from the interface data, and extracting first code features from the first business code; encoding the voice description data to obtain voice timing features, and extracting second code features from the second business code; encoding the text description data to obtain text semantic features, and extracting third code features from the third business code; extracting the association relationship between the interface layout features and the voice timing features and the text semantic features to obtain first association features, extracting the association relationship between the voice timing features and the interface layout features and the text semantic features to obtain second association features, and extracting the association relationship between the text semantic features and the voice timing features and the interface layout features to obtain third association features; marking the first code features to the interface layout features and the first association features, marking the second code features to the voice timing features and the second association features, and marking the third code features to the text semantic features and the third association features.

[0011] As an optional implementation mode, the adjusting the initial business implementation code to obtain the target business implementation code includes: transmitting the initial business implementation code to a code optimization and checking component by the target code generation model, wherein the code optimization and checking component is used to perform syntax checking and syntax adjustment on the initial business implementation code to obtain a reference business implementation code, and perform performance detection and performance adjustment on the reference business implementation code to obtain the target business implementation code whose code performance is higher than a target performance threshold; and the code optimization and checking component outputs the target business implementation code.

[0012] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a device for generating a business implementation code is provided. The device comprises: a receiving module, which is used to receive a code generation request initiated by a target account, wherein the code generation request is used to request the generation of a business implementation code for implementing a business function indicated by target business data, and the data modality of the target business data includes at least one of the following: image modality, audio modality and text modality; a generating module, which is used to respond to the code generation request and generate an initial business implementation code of the target business data through a target code generation model, wherein the target code generation model is obtained by training the initial code generation model using multimodal business data with code samples annotated, and the multimodal business data includes: business image data, business audio data and business text data; an adjusting module, which is used to adjust the initial business implementation code to obtain a target business implementation code; and a sending module, which is used to send the target business implementation code to the target account.

[0013] As an optional implementation, the generation module includes: an extraction unit, used to extract code condition information from the code generation request, wherein the code condition information is used to indicate the code parameter configuration and code language proposed by the target account for the generated code; an input unit, used to input the code condition information and the target business data into the target code generation model; and an acquisition unit, used to acquire the initial business implementation code output by the target code generation model that meets the code condition information.

[0014] As an optional implementation, the generation module also includes: a construction unit, used to use the business code and business data of the implemented business to construct multimodal business data annotated with code samples to obtain a training data set; a training unit, used to use the training data set to train the initial code generation model to obtain the target code generation model.

[0015] As an optional implementation, the construction unit is also used to: collect business codes and business data of implemented businesses; extract business data of multiple data modes from the business data, and extract the code samples corresponding to each business data from the business code, to obtain a first business code corresponding to the business image data in the image mode, a second business code corresponding to the business audio data in the audio mode, and a third business code corresponding to the business text data in the text mode, wherein the multiple data modes include: the image mode, the audio mode and the text mode; annotate the first business code to the business image data, annotate the second business code to the business audio data, and annotate the third business code to the business text data to obtain the training data set.

[0016] Optionally, the construction unit is also used to: extract interface data from the business data, and extract the business implementation code of the interface data from the business code, to obtain a first business code corresponding to the business image data in the image mode; extract voice description data from the business data, and extract the business implementation code of the voice description data from the business code, to obtain a second business code corresponding to the business audio data in the audio mode; extract text description data from the business data, and extract the business implementation code of the text description data from the business code, to obtain a third business code corresponding to the business text data in the text mode.

[0017] Optionally, the construction unit is also used to: extract interface layout features from the interface data, and extract first code features from the first business code; encode the voice description data to obtain voice timing features, and extract second code features from the second business code; encode the text description data to obtain text semantic features, and extract third code features from the third business code; extract the association relationship between the interface layout features and the voice timing features and the text semantic features to obtain first association features, extract the association relationship between the voice timing features and the interface layout features and the text semantic features to obtain second association features, and extract the association relationship between the text semantic features and the voice timing features and the interface layout features to obtain third association features; mark the first code features to the interface layout features and the first association features, mark the second code features to the voice timing features and the second association features, and mark the third code features to the text semantic features and the third association features.

[0018] As an optional implementation, the adjustment module includes: a transmission unit, used to transmit the initial business implementation code to the code optimization and checking component by the target code generation model, wherein the code optimization and checking component is used to perform syntax checking and syntax adjustment on the initial business implementation code to obtain a reference business implementation code, and perform performance detection and performance adjustment on the reference business implementation code to obtain the target business implementation code whose code performance is higher than a target performance threshold; an output unit, used to output the target business implementation code by the code optimization and checking component.

[0019] In an embodiment of the present application, a code generation request for requesting generation of a business implementation code for implementing a business function indicated by target business data is received from a target account, target business data including at least one of an image modality, an audio modality, and a text modality is received as a basis for generating the business implementation code, and then an initial business implementation code for the target business data is generated by a target code generation model. Because the target code generation model is obtained by training the initial code generation model using multimodal business data annotated with code samples, and the multimodal business data includes business image data, business audio data, and business text data, the initial business implementation code generated using the target code generation model has learned the code corresponding to the multimodal data, and can improve the quality of the generated code by combining different data modalities and the relationship between different modal data. After that, the initial business implementation code will be adjusted, and the adjusted target business implementation code will be sent to the target account, so as to achieve the purpose of generating high-quality business implementation code, thereby achieving the technical effect of improving the generation efficiency of the business implementation code, and thus solving the technical problem of low generation efficiency of the business implementation code caused by the code generation model only generating code for a single modality data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 A hardware structure block diagram of a computer terminal for implementing a method for generating a business implementation code is shown;

[0022] Figure 2 It is a flowchart of a method for generating a business implementation code according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a code generation method based on multimodal data according to an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of a device for generating a service implementation code according to an embodiment of the present application;

[0025] Figure 5 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] It should be noted that the collected information (including but not limited to user device information, user personal information, code information of implemented services, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0029] Example 1

[0030] According to an embodiment of the present application, an embodiment of a method for generating a business implementation code is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating a business implementation code is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0032] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method for generating the business implementation code in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for generating the business implementation code described above is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0036] Under the above operating environment, this application provides Figure 2 The method for generating the business implementation code shown. Figure 2 It is a flowchart of a method for generating business implementation code according to Example 1 of the present application.

[0037] Step S201, receiving a code generation request initiated by a target account, wherein the code generation request is used to request generation of a business implementation code for implementing a business function indicated by target business data, and the data modality of the target business data includes at least one of the following: image modality, audio modality, and text modality;

[0038] Step S202, in response to the code generation request, generating initial business implementation code for the target business data through a target code generation model, wherein the target code generation model is obtained by training an initial code generation model using multimodal business data annotated with code samples, and the multimodal business data includes: business image data, business audio data, and business text data;

[0039] Step S203, adjusting the initial service implementation code to obtain a target service implementation code;

[0040] Step S204: sending the target service implementation code to the target account.

[0041] The method for generating a business implementation code provided in an embodiment of the present application receives a code generation request initiated by a target account for requesting generation of a business implementation code for implementing a business function indicated by target business data, receives target business data including at least one of an image modality, an audio modality, and a text modality as a basis for generating the business implementation code, and then generates an initial business implementation code for the target business data through a target code generation model. Because the target code generation model is obtained by training the initial code generation model using multimodal business data with annotated code samples, and the multimodal business data includes business image data, business audio data, and business text data, the initial business implementation code generated using the target code generation model has learned the code corresponding to the multimodal data, and can improve the quality of the generated code by combining different data modalities and the relationship between different modal data. Afterwards, the initial business implementation code will be adjusted, and the adjusted target business implementation code will be sent to the target account, thereby achieving the purpose of generating high-quality business implementation code, solving the technical problem of low business implementation code generation efficiency caused by the code generation model only generating code for a single modality data, thereby achieving the technical effect of improving the business implementation code generation efficiency.

[0042] In the embodiment provided in step S201, the code generation request includes but is not limited to a request to generate interface code based on user requirements described in natural language, or a request to generate function code according to audio instructions, or a request to generate corresponding front-end code according to an interface design drawing, or a request to automatically generate an algorithm implementation code according to a mathematical description or text description of the algorithm, etc.

[0043] Optionally, in this embodiment, the target business data provided by the code generation request includes one or more of image modality data, audio modality data, and text modality data. In the case where the target business data includes multiple modality data, including but not limited to assigning different weights to data of different modalities according to the indicated business function, i.e., the business function to be implemented, to reflect their importance in the generation of the implementation code of the business function. For example, visual information may be more critical to the code generation of the interface design, while text description may be more important to the code generation of the algorithm or logic part.

[0044] Optionally, in an embodiment of the present application, data in image modality may be but is not limited to data displayed in an image display mode, data in audio modality may be but is not limited to data displayed in an audio display mode, and data in text modality may be but is not limited to data displayed in a text display mode, that is, data in image modality may actually be converted into data in audio modality, and / or data in text modality, and the substantive content represented by the data before and after the conversion is the same, but the ability to display the substantive content before and after the conversion may be different.

[0045] In the embodiment provided in step S202, the multimodal business data annotated with the code sample may be, but is not limited to, modal data annotated with the code sample and associations between different modal data annotated with the code sample.

[0046] Optionally, in an embodiment of the present application, generating the initial business implementation code for the target business data through the target code generation model includes but is not limited to obtaining the code generation preference information of the target account based on the historical code generation records of the target account, wherein the code generation preference information is used to indicate the code language and code logic preferred by the target account; inputting the code generation preference information and the target business data into the target code generation model to obtain the business implementation code output by the target code generation model.

[0047] In the embodiment provided in step S203, adjusting the initial business implementation code includes but is not limited to adjusting the syntax logic, naming form, and code placement format of the code.

[0048] In the embodiment provided in step S204, sending the target business implementation code to the target account may include, but is not limited to, adding comments to the target business implementation code, and sending the target business implementation code carrying the comments to the target account.

[0049] Optionally, in an embodiment of the present application, multimodal data related to the target business implementation code may also be sent when the target business implementation code is sent, so as to inspire the user to enrich the provided target business data.

[0050] Optionally, in the method for generating a business implementation code provided in an embodiment of the present application, the step of generating the initial business implementation code of the target business data by using a target code generation model includes:

[0051] Extracting code condition information from the code generation request, wherein the code condition information is used to indicate the code parameter configuration and code language proposed by the target account for the generated code;

[0052] Inputting the code condition information and the target business data into the target code generation model;

[0053] The initial business implementation code output by the target code generation model and conforming to the code condition information is obtained.

[0054] Optionally, in this embodiment, the code condition information may be, but is not limited to, carried by the user in the code generation request, or determined according to the requirements of the business function.

[0055] Through the above content, the code condition information and target business data determined from the code generation request are used as the input of the target code generation model, so that the target code generation model outputs the business implementation code that meets the code condition information, and extracts richer user demand information from the existing information as the basis for code generation, so that the generated code is closer to user needs and the code generation efficiency is improved.

[0056] Optionally, in the method for generating a service implementation code provided in an embodiment of the present application, before inputting the code condition information and the target service data into the target code generation model, the method further includes:

[0057] Use the business code and business data of the implemented business to construct multimodal business data with annotated code samples to obtain a training data set;

[0058] The initial code generation model is trained using the training data set to obtain the target code generation model.

[0059] Optionally, in this embodiment, including but not limited to limiting the age of the business code and business data of the business used, using more recent business codes and business data as the basis for obtaining a training data set can make the code output by the target code generation model more consistent with the current common code logic, easier for users to understand, and avoid incompatibility issues.

[0060] Through the above content, the initial code generation model is retrained with multimodal business data annotated with code samples constructed through the business code and business data of the implemented business. The training range of the code generation model is wider, and the obtained target code generation model will have a better code generation effect, thereby improving the generation efficiency of the business implementation code.

[0061] Optionally, in the method for generating the service implementation code provided in the embodiment of the present application, the multimodal service data annotated with the code samples is constructed using the service code and service data of the implemented service to obtain the training data set, including:

[0062] Collect business codes and business data of implemented businesses;

[0063] Extracting business data of multiple data modalities from the business data, and extracting the code samples corresponding to each business data from the business code, to obtain a first business code corresponding to the business image data in the image mode, a second business code corresponding to the business audio data in the audio mode, and a third business code corresponding to the business text data in the text mode, wherein the multiple data modalities include: the image mode, the audio mode, and the text mode;

[0064] The first service code is marked on the service image data, the second service code is marked on the service audio data, and the third service code is marked on the service text data to obtain the training data set.

[0065] Optionally, in this embodiment, different business data may correspond to the same code sample, for example, business image data 1 , business audio data 2 , and business text data 3 may, but are not limited to, all correspond to business code 1 .

[0066] Optionally, in this embodiment, multiple data modalities include image modality, audio modality and text modality. When there is data in other modalities, the data in other modalities can be converted into data in image modality, audio modality and text modality, but is not limited to it, and then the business code corresponding to the data in other modalities is marked on the modal data obtained by the above conversion.

[0067] Through the above content, data of different modalities and their corresponding codes are extracted, and the extracted codes are annotated to data of different modalities, obtaining a training set containing multimodal data and their corresponding code implementations. This enriches the training set of the code generation model from the perspective of basic data types, enabling the target code generation model to learn solutions for generating code from different modalities, broadening the range of code generation scenarios that the target code generation model can cope with and improving the code generation efficiency.

[0068] Optionally, in the method for generating a business implementation code provided in an embodiment of the present application, extracting business data of multiple data modalities from the business data, and extracting the code sample corresponding to each business data from the business code, to obtain a first business code corresponding to the business image data in the image modality, a second business code corresponding to the business audio data in the audio modality, and a third business code corresponding to the business text data in the text modality, includes:

[0069] Extracting interface data from the business data, and extracting a business implementation code of the interface data from the business code, to obtain a first business code corresponding to the business image data in the image modality;

[0070] Extracting voice description data from the service data, and extracting a service implementation code of the voice description data from the service code, to obtain a second service code corresponding to the service audio data in the audio mode;

[0071] The text description data is extracted from the business data, and the business implementation code of the text description data is extracted from the business code to obtain a third business code corresponding to the business text data in the text mode.

[0072] Through the above content, different data and the business implementation codes corresponding to different data are extracted respectively, which provides a basis for creating multimodal business data with annotated code samples.

[0073] Optionally, in the method for generating a service implementation code provided in an embodiment of the present application, the step of marking the first service code to the service image data, marking the second service code to the service audio data, and marking the third service code to the service text data to obtain the training data set includes:

[0074] Extracting interface layout features from the interface data, and extracting first code features from the first business code; encoding the voice description data to obtain voice timing features, and extracting second code features from the second business code; encoding the text description data to obtain text semantic features, and extracting third code features from the third business code;

[0075] Extracting the association relationship between the interface layout feature and the speech timing feature and the text semantic feature to obtain a first association feature, extracting the association relationship between the speech timing feature and the interface layout feature and the text semantic feature to obtain a second association feature, and extracting the association relationship between the text semantic feature and the speech timing feature and the interface layout feature to obtain a third association feature;

[0076] The first code feature is annotated to the interface layout feature and the first associated feature, the second code feature is annotated to the speech timing feature and the second associated feature, and the third code feature is annotated to the text semantic feature and the third associated feature.

[0077] Optionally, in the present embodiment, extracting interface layout features from interface data includes but is not limited to extracting interface layout features such as button positions, color distribution, etc. using a convolutional neural network (CNN); encoding voice description data to obtain voice timing features includes but is not limited to encoding the voice description using a recurrent neural network (RNN) or a Transformer model to extract its timing features; encoding text description data to obtain text semantic features includes but is not limited to encoding the text description using a Transformer model to extract its semantic features; extracting first code features from the first business code, extracting second business code features from the second business code, and extracting third code features from the third business code includes but is not limited to extracting feature values ​​such as code structure, syntax, semantics, and complexity from the code through a lexical analysis algorithm, a grammatical analysis algorithm, a semantic analysis algorithm, and the like.

[0078] Through the above content, different data features of different data are extracted, and the correlation features between different data features are also extracted. While the code features are labeled on the data features, the code is also labeled on the correlation features. Therefore, the training set obtained for the initial code generation model also takes into account the correlation relationship between different modal data, expands the training content of the model, and can improve the quality of the code generated by the model.

[0079] Optionally, in the method for generating a service implementation code provided in an embodiment of the present application, adjusting the initial service implementation code to obtain a target service implementation code includes:

[0080] The target code generation model transmits the initial business implementation code to a code optimization and checking component, wherein the code optimization and checking component is used to perform syntax checking and syntax adjustment on the initial business implementation code to obtain a reference business implementation code, and to perform performance detection and performance adjustment on the reference business implementation code to obtain the target business implementation code whose code performance is higher than a target performance threshold;

[0081] The target business implementation code is output by the code optimization and inspection component.

[0082] Through the above content, further inspection and adjustment of the initial business implementation code can further improve the quality of the code provided to the target account and improve the efficiency of generating the business implementation code.

[0083] As an optional implementation, this application also provides an intelligent code generation method based on a multimodal model, which combines different types of data (such as text, images, audio, etc.) and uses their complementarity to enable the machine to better understand and generate code. This method can effectively solve the current problem of low quality and accuracy of intelligently generated code, and can also provide more comprehensive and diverse programming support for design developers.

[0084] More specifically, the intelligent code generation method based on the multimodal model is a tool that can extract information from multiple modalities (such as text, images, audio, etc.) and automatically generate code. The multimodal model of the code field is formed through training of multimodal data and its associated relationship data. Subsequently, based on the interface diagram or audio information submitted by the user, the implementation code with a high degree of similarity can be automatically generated.

[0085] The implementation of this method includes the following steps:

[0086] Step S1, extracting training data sets such as interfaces and their implementation codes, components and their implementation codes, audio and their implementation codes from software products that have realized business functions and can extract data;

[0087] Step S2, preprocessing the training data set, extracting the feature values ​​of the interface, audio, code and other data, and annotating the association relationship (such as the feature value of interface 1 is associated with the feature value of code 1, and the feature value of interface 2 is associated with the feature value of code 2);

[0088] Step S3, conducting multimodal model training based on the preprocessed training data, and conducting multiple rounds of parameter adjustment training in combination with actual training results;

[0089] Step S4, upload the target interface or audio information that needs to be implemented in the actual demand scenario to the multimodal model, and the model automatically generates code that meets the actual needs based on the feature values ​​of the request information.

[0090] Figure 3 is a schematic diagram of a code generation method based on multimodal data according to an embodiment of the present application. Figure 3 As shown, the tool mainly includes the following core components:

[0091] Multimodal data preprocessing component: This component is responsible for obtaining multimodal data from different data sources and converting it into a unified format. For text data, it is necessary to perform word segmentation, part-of-speech tagging, and stop word removal; for image data, it is necessary to perform cropping, scaling, and normalization; for audio data, it is necessary to perform noise reduction and sampling rate conversion.

[0092] Feature extraction component: This component is responsible for extracting feature values ​​from preprocessed data, using convolutional neural networks (CNN) to extract interface layout features, such as button positions, color distribution, etc.; using recurrent neural networks (RNN) or Transformer models to encode voice instructions and extract their temporal features; using Transformer models to encode text descriptions and extract their semantic features; extracting feature values ​​such as code structure, syntax, semantics, and complexity from the code through lexical analysis algorithms, syntax analysis algorithms, and semantic analysis algorithms. At the same time, the association between different feature values ​​can be annotated.

[0093] Model training component: Based on the extracted features, this component uses machine learning and deep learning algorithms to train the model to identify different modal information and their correlations.

[0094] Code generation component: Based on the trained model and user requirements (generating specified graphical interfaces, components or audio), this component automatically generates corresponding codes. The generated codes can be in any programming language.

[0095] Code optimization and checking component: The generated code needs to be optimized and checked to ensure that it can run normally and has efficient performance. This component can perform syntax checking, performance evaluation and other operations on the generated code to ensure its quality and correctness.

[0096] Visual interface: For the convenience of users, the tool also includes a visual interface, through which users can upload multimodal data and its implementation code, configure the parameters of generated code, view the generated code, etc.

[0097] Through the intelligent code generation technology based on the multimodal model shown above, the following functions can be achieved:

[0098] 1. Improve code generation efficiency: Multimodal models can use multiple different forms of data input at the same time, such as interface, graphics, audio, etc., so that the machine can understand and process information more comprehensively. In this way, the system can generate high-quality code faster, and the coding efficiency has been further improved compared to the current method of generating code through multiple rounds of natural language interaction.

[0099] 2. Enhance the correctness and stability of the code: Multimodal models can enable machines to better understand the context and context of information through data input in a variety of different forms. In the code generation process, this means that errors and exceptions can be reduced, and the correctness and stability of the code can be improved.

[0100] 3. Realize a wider range of application scenarios: The multimodal model can flexibly select and combine different modal data according to different application scenarios and requirements to achieve more intelligent and flexible code generation.

[0101] In summary, the method provided in this application combines the advantages of multimodal information, improves the quality of automatically generated code, and improves the R&D efficiency of developers.

[0102] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0103] Example 2

[0104] The embodiment of the present application also provides a device for generating a business implementation code. It should be noted that the device for generating a business implementation code in the embodiment of the present application can be used to execute the method for generating a business implementation code provided in the embodiment of the present application. The device for generating a business implementation code provided in the embodiment of the present application is introduced below.

[0105] According to an embodiment of the present application, a device for implementing the above-mentioned method for generating a service implementation code is also provided, such as Figure 4 As shown, the device comprises:

[0106] The receiving module 401 is used to receive a code generation request initiated by a target account, wherein the code generation request is used to request generation of a business implementation code for implementing a business function indicated by target business data, and the data modality of the target business data includes at least one of the following: image modality, audio modality, and text modality;

[0107] A generation module 402 is used to respond to the code generation request and generate initial business implementation code for the target business data through a target code generation model, wherein the target code generation model is obtained by training an initial code generation model using multimodal business data annotated with code samples, and the multimodal business data includes: business image data, business audio data, and business text data;

[0108] An adjustment module 403 is used to adjust the initial service implementation code to obtain a target service implementation code;

[0109] The sending module 404 is used to send the target service implementation code to the target account.

[0110] The device for generating a business implementation code provided in an embodiment of the present application receives a code generation request initiated by a target account for requesting generation of a business implementation code for implementing a business function indicated by target business data, receives target business data including at least one of an image modality, an audio modality, and a text modality as a basis for generating the business implementation code, and then generates an initial business implementation code for the target business data through a target code generation model. Because the target code generation model is obtained by training the initial code generation model using multimodal business data with annotated code samples, and the multimodal business data includes business image data, business audio data, and business text data, the initial business implementation code generated using the target code generation model has learned the code corresponding to the multimodal data, and can improve the quality of the generated code by combining different data modalities and the relationship between different modal data. Afterwards, the initial business implementation code will be adjusted, and the adjusted target business implementation code will be sent to the target account, thereby achieving the purpose of generating high-quality business implementation code, solving the technical problem of low generation efficiency of the business implementation code caused by the code generation model only generating code for a single modality data, thereby achieving the technical effect of improving the generation efficiency of the business implementation code.

[0111] Optionally, in the device for generating business implementation code provided in an embodiment of the present application, the generation module includes: an extraction unit, used to extract code condition information from the code generation request, wherein the code condition information is used to indicate the code parameter configuration and code language proposed by the target account for the generated code; an input unit, used to input the code condition information and the target business data into the target code generation model; and an acquisition unit, used to acquire the initial business implementation code output by the target code generation model that meets the code condition information.

[0112] Optionally, in the device for generating business implementation code provided in an embodiment of the present application, the generation module also includes: a construction unit, used to use the business code and business data of the implemented business to construct multimodal business data annotated with code samples to obtain a training data set; and a training unit, used to use the training data set to train the initial code generation model to obtain the target code generation model.

[0113] Optionally, in the device for generating the business implementation code provided in the embodiment of the present application, the construction unit is further used to: collect the business code and business data of the implemented business; extract the business data of multiple data modes from the business data, and extract the code sample corresponding to each business data from the business code, to obtain a first business code corresponding to the business image data in the image mode, a second business code corresponding to the business audio data in the audio mode, and a third business code corresponding to the business text data in the text mode, wherein the multiple data modes include: the image mode, the audio mode and the text mode; annotate the first business code to the business image data, annotate the second business code to the business audio data, and annotate the third business code to the business text data to obtain the training data set.

[0114] Optionally, in the device for generating the business implementation code provided in the embodiment of the present application, the construction unit is also used to: extract interface data from the business data, and extract the business implementation code of the interface data from the business code, to obtain a first business code corresponding to the business image data in the image mode; extract voice description data from the business data, and extract the business implementation code of the voice description data from the business code, to obtain a second business code corresponding to the business audio data in the audio mode; extract text description data from the business data, and extract the business implementation code of the text description data from the business code, to obtain a third business code corresponding to the business text data in the text mode.

[0115] Optionally, in the device for generating the business implementation code provided in the embodiment of the present application, the construction unit is also used to: extract interface layout features from the interface data, and extract first code features from the first business code; encode the voice description data to obtain voice timing features, and extract second code features from the second business code; encode the text description data to obtain text semantic features, and extract third code features from the third business code; extract the association relationship between the interface layout features and the voice timing features and the text semantic features respectively to obtain first association features, extract the association relationship between the voice timing features and the interface layout features and the text semantic features respectively to obtain second association features, and extract the association relationship between the text semantic features and the voice timing features and the interface layout features respectively to obtain third association features; annotate the first code features to the interface layout features and the first association features, annotate the second code features to the voice timing features and the second association features, and annotate the third code features to the text semantic features and the third association features.

[0116] Optionally, in the device for generating business implementation code provided in an embodiment of the present application, the adjustment module includes: a transmission unit, used to transmit the initial business implementation code to a code optimization and checking component by the target code generation model, wherein the code optimization and checking component is used to perform syntax checking and syntax adjustment on the initial business implementation code to obtain a reference business implementation code, and perform performance testing and performance adjustment on the reference business implementation code to obtain the target business implementation code whose code performance is higher than a target performance threshold; an output unit, used to output the target business implementation code by the code optimization and checking component.

[0117] It should be noted that the receiving module 401, generating module 402, adjusting module 403 and sending module 104 correspond to steps S201 to S204 in Example 1, and the examples and application scenarios implemented by each module and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules can also be run in the computer terminal 10 provided in Example 1 as part of the device.

[0118] Example 3

[0119] An embodiment of the present application may provide an electronic device, Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (only one is shown) processor 502, memory 504, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0120] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0121] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive a code generation request initiated by a target account, wherein the code generation request is used to request the generation of a business implementation code that implements the business function indicated by the target business data, and the data modality of the target business data includes at least one of the following: image modality, audio modality and text modality; in response to the code generation request, generate an initial business implementation code for the target business data through a target code generation model, wherein the target code generation model is obtained by training an initial code generation model using multimodal business data with code samples annotated, and the multimodal business data includes: business image data, business audio data and business text data; adjust the initial business implementation code to obtain a target business implementation code; and send the target business implementation code to the target account.

[0122] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: extract code condition information from the code generation request, wherein the code condition information is used to indicate the code parameter configuration and code language proposed by the target account for the generated code; input the code condition information and the target business data into the target code generation model; obtain the initial business implementation code output by the target code generation model that meets the code condition information.

[0123] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: use the business code and business data of the implemented business to construct multimodal business data with labeled code samples to obtain a training data set; use the training data set to train the initial code generation model to obtain the target code generation model.

[0124] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: collect the business code and business data of the implemented business; extract the business data of multiple data modes from the business data, and extract the code sample corresponding to each business data from the business code to obtain the first business code corresponding to the business image data in the image mode, the second business code corresponding to the business audio data in the audio mode, and the third business code corresponding to the business text data in the text mode, wherein the multiple data modes include: the image mode, the audio mode and the text mode; mark the first business code to the business image data, mark the second business code to the business audio data, and mark the third business code to the business text data to obtain the training data set.

[0125] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: extract interface data from the business data, and extract the business implementation code of the interface data from the business code to obtain the first business code corresponding to the business image data in the image mode; extract voice description data from the business data, and extract the business implementation code of the voice description data from the business code to obtain the second business code corresponding to the business audio data in the audio mode; extract text description data from the business data, and extract the business implementation code of the text description data from the business code to obtain the third business code corresponding to the business text data in the text mode.

[0126] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: extracting interface layout features from the interface data, and extracting first code features from the first business code; encoding the voice description data to obtain voice timing features, and extracting second code features from the second business code; encoding the text description data to obtain text semantic features, and extracting third code features from the third business code; extracting the association relationship between the interface layout features and the voice timing features and the text semantic features to obtain first association features, extracting the association relationship between the voice timing features and the interface layout features and the text semantic features to obtain second association features, and extracting the association relationship between the text semantic features and the voice timing features and the interface layout features to obtain third association features; marking the first code features to the interface layout features and the first association features, marking the second code features to the voice timing features and the second association features, and marking the third code features to the text semantic features and the third association features.

[0127] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: the target code generation model transmits the initial business implementation code to the code optimization and checking component, wherein the code optimization and checking component is used to perform syntax checking and syntax adjustment on the initial business implementation code to obtain a reference business implementation code, and to perform performance detection and performance adjustment on the reference business implementation code to obtain the target business implementation code whose code performance is higher than the target performance threshold; the code optimization and checking component outputs the target business implementation code.

[0128] By adopting the embodiment of the present application, a solution for generating business implementation code is provided. By receiving a code generation request initiated by a target account for requesting to generate a business implementation code for a business function indicated by target business data, receiving target business data including at least one of an image modality, an audio modality, and a text modality as a basis for generating the business implementation code, and then generating the initial business implementation code of the target business data through a target code generation model, because the target code generation model is obtained by training the initial code generation model using multimodal business data with annotated code samples, and the multimodal business data includes business image data, business audio data, and business text data, then the initial business implementation code generated using the target code generation model is the code corresponding to the multimodal data learned, and can improve the quality of the generated code by combining different data modalities and the relationship between different modal data, and then the initial business implementation code will be adjusted, and the adjusted target business implementation code will be sent to the target account, so as to achieve the purpose of generating high-quality business implementation code, thereby achieving the technical effect of improving the generation efficiency of the business implementation code, and thus solving the technical problem of low generation efficiency of the business implementation code caused by the code generation model only generating code for a single modal data.

[0129] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 5 The structure of the electronic device is not limited. Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5 Different configurations are shown.

[0130] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0131] Example 4

[0132] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for generating the service implementation code provided in the first embodiment.

[0133] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0134] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the method for generating business implementation code.

[0135] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0136] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0141] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating a business implementation code, characterized in that: include: Receiving a code generation request initiated by a target account, wherein the code generation request is used to request generation of a business implementation code for implementing a business function indicated by target business data, wherein the data modality of the target business data includes at least one of the following: image modality, audio modality, and text modality; In response to the code generation request, generating initial business implementation code for the target business data through a target code generation model, wherein the target code generation model is obtained by training an initial code generation model using multimodal business data annotated with code samples, and the multimodal business data includes: business image data, business audio data, and business text data; Adjusting the initial business implementation code to obtain a target business implementation code; The target service implementation code is sent to the target account.

2. The method according to claim 1, characterized in that The generating the initial business implementation code of the target business data by using the target code generation model includes: Extracting code condition information from the code generation request, wherein the code condition information is used to indicate the code parameter configuration and code language proposed by the target account for the generated code; Inputting the code condition information and the target business data into the target code generation model; The initial business implementation code output by the target code generation model and conforming to the code condition information is obtained.

3. The method according to claim 2, characterized in that Before inputting the code condition information and the target service data into the target code generation model, the method further includes: Use the business code and business data of the implemented business to construct multimodal business data with annotated code samples to obtain a training data set; The initial code generation model is trained using the training data set to obtain the target code generation model.

4. The method according to claim 3, characterized in that The multimodal business data with code samples annotated are constructed using the business code and business data of the implemented business to obtain a training data set, including: Collect business codes and business data of implemented businesses; Extracting business data of multiple data modalities from the business data, and extracting the code samples corresponding to each business data from the business code, to obtain a first business code corresponding to the business image data in the image mode, a second business code corresponding to the business audio data in the audio mode, and a third business code corresponding to the business text data in the text mode, wherein the multiple data modalities include: the image mode, the audio mode, and the text mode; The first service code is marked on the service image data, the second service code is marked on the service audio data, and the third service code is marked on the service text data to obtain the training data set.

5. The method according to claim 4, characterized in that The extracting of the business data of multiple data modalities from the business data, and extracting the code samples corresponding to the respective business data from the business codes, to obtain a first business code corresponding to the business image data in the image modality, a second business code corresponding to the business audio data in the audio modality, and a third business code corresponding to the business text data in the text modality, includes: Extracting interface data from the business data, and extracting a business implementation code of the interface data from the business code, to obtain a first business code corresponding to the business image data in the image modality; Extracting voice description data from the service data, and extracting a service implementation code of the voice description data from the service code, to obtain a second service code corresponding to the service audio data in the audio mode; The text description data is extracted from the business data, and the business implementation code of the text description data is extracted from the business code to obtain a third business code corresponding to the business text data in the text mode.

6. The method according to claim 5, characterized in that The step of marking the first service code to the service image data, marking the second service code to the service audio data, and marking the third service code to the service text data to obtain the training data set includes: Extracting interface layout features from the interface data, and extracting first code features from the first business code; encoding the voice description data to obtain voice timing features, and extracting second code features from the second business code; encoding the text description data to obtain text semantic features, and extracting third code features from the third business code; Extracting the association relationship between the interface layout feature and the speech timing feature and the text semantic feature to obtain a first association feature, extracting the association relationship between the speech timing feature and the interface layout feature and the text semantic feature to obtain a second association feature, and extracting the association relationship between the text semantic feature and the speech timing feature and the interface layout feature to obtain a third association feature; The first code feature is annotated to the interface layout feature and the first associated feature, the second code feature is annotated to the speech timing feature and the second associated feature, and the third code feature is annotated to the text semantic feature and the third associated feature.

7. The method according to claim 1, characterized in that The adjusting the initial service implementation code to obtain the target service implementation code includes: The target code generation model transmits the initial business implementation code to a code optimization and checking component, wherein the code optimization and checking component is used to perform syntax checking and syntax adjustment on the initial business implementation code to obtain a reference business implementation code, and to perform performance detection and performance adjustment on the reference business implementation code to obtain the target business implementation code whose code performance is higher than a target performance threshold; The target business implementation code is output by the code optimization and inspection component.

8. A device for generating a service implementation code, characterized in that: include: A receiving module, configured to receive a code generation request initiated by a target account, wherein the code generation request is used to request generation of a business implementation code for implementing a business function indicated by target business data, wherein the data modality of the target business data includes at least one of the following: image modality, audio modality, and text modality; A generation module, configured to respond to the code generation request and generate initial business implementation code for the target business data through a target code generation model, wherein the target code generation model is obtained by training an initial code generation model using multimodal business data annotated with code samples, and the multimodal business data includes: business image data, business audio data, and business text data; An adjustment module, used for adjusting the initial business implementation code to obtain a target business implementation code; A sending module is used to send the target business implementation code to the target account.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the method for generating a service implementation code according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 7 when running.

Images