Data processing method, visual draft processing method, system and electronic device

Abstract

The embodiments of the present application provide a data processing method, a method for processing a visual draft, a system, and an electronic device. The method includes: determining the layout information of the image based on the image file; processing the image based on the layout information to determine the local image with component functions in the image; and obtaining first description information for generating program code based on the layout information and the local image with component functions. The technical method provided in the present application can automatically generate the first description information with high accuracy; in addition, the first description information can be used to obtain the program code of the image, providing a technical basis for providing automatic code generation services online.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a visual draft processing method, a system, and an electronic device. Background Art

[0002] Automatically generating program code from visual images is an effective method for improving system and program development efficiency and shortening development cycles. Currently, this automatic code generation process often uses visual images designed by visual designers as input, and uses object detection to identify UI (User Interface) components. This method requires identifying both the component type and its location within the visual image, resulting in low accuracy and impracticality for large-scale application. Summary of the Invention

[0003] The present application provides a data processing method, a method for processing visual drafts, a system and an electronic device that solve the above-mentioned problems or at least partially solve the above-mentioned problems.

[0004] In one embodiment of the present application, a data processing method is provided. The method includes:

[0005] Determining layout information of the picture according to the picture file;

[0006] Processing the image based on the layout information to determine a partial image in the image having a component function;

[0007] First description information for generating program code is obtained according to the layout information and the local diagram having component functions.

[0008] In one embodiment of the present application, a data processing system is provided. The system includes:

[0009] The client is used to send image files to the server;

[0010] The server is used to determine the layout information of the image based on the image file of the image; based on the layout information, the image is processed to determine the local image with component function in the image; based on the layout information and the local image with component function, the first description information for generating program code is obtained.

[0011] In another embodiment of the present application, a method for processing a visual draft is provided. The method includes:

[0012] Determining layout information of the visual draft according to the visual draft file;

[0013] Based on the layout information, the page diagram corresponding to the visual draft is processed to determine a partial diagram having component functions in the page diagram;

[0014] First description information for generating program code is obtained according to the layout information and the local diagram having component functions.

[0015] In another embodiment of the present application, a system for processing a visual draft is provided. The system includes:

[0016] The client is used to send the visual draft file to the server;

[0017] The server is used to determine the layout information of the visual draft based on the visual draft file; based on the layout information, process the page diagram corresponding to the visual draft to obtain a local diagram with component functions in the page diagram; based on the layout information and the local diagram with component functions, obtain the first description information for generating program code.

[0018] In one embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein:

[0019] The memory is used to store programs;

[0020] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0021] Determining layout information of the picture according to the picture file;

[0022] Processing the image based on the layout information to determine a partial image in the image having a component function;

[0023] First description information for generating program code is obtained according to the layout information and the local diagram having component functions.

[0024] In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein:

[0025] The memory is used to store programs;

[0026] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0027] Determining layout information of the visual draft according to the visual draft file;

[0028] Based on the layout information, the page diagram corresponding to the visual draft is processed to determine a partial diagram having component functions in the page diagram;

[0029] First description information for generating program code is obtained according to the layout information and the local diagram having component functions.

[0030] A technical solution provided by an embodiment of the present application determines the layout information of a picture (or a page diagram corresponding to a visual draft) based on the picture file (or visual draft file) of the picture, and then processes the picture (or page diagram) based on the layout information to obtain a partial diagram with component functions in the picture (or page diagram), and then obtains the first description information for generating program code based on the layout information and the partial diagram with component functions. In the above process, the partial diagram with component functions in the picture is obtained by processing the picture (or page diagram) based on the layout information, so the position information of the partial diagram with component functions in the picture (or page diagram) can be accurately obtained; as long as the recognition accuracy of the partial diagram with component functions is guaranteed, the generated program code must be accurate; and the image recognition accuracy of the current image recognition technology can still be effectively guaranteed. It can be seen that compared with the existing solutions that need to identify both components and component positions, the solution provided by the embodiment of the present application is more accurate and suitable for large-scale use. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 A schematic diagram illustrating the principle of image recognition of a page image corresponding to a visual draft provided in one embodiment of the present application;

[0033] Figure 2 A schematic diagram of the structure of a data processing system provided in one embodiment of the present application;

[0034] Figure 3 A flowchart of a data processing method provided in one embodiment of the present application;

[0035] Figure 4a A schematic diagram showing layout information of an image in an image file and at least one partial image obtained based on the layout information, provided in one embodiment of the present application;

[0036] Figure 4b A schematic diagram of a principle for determining the category to which the content of at least one partial image belongs, provided in an embodiment of the present application;

[0037] Figure 5A schematic diagram of two partial images of the same component category in a picture provided in an embodiment of the present application, which are in a nested relationship;

[0038] Figure 6 A schematic diagram of a visual editing page provided in one embodiment of the present application;

[0039] Figure 7 A schematic diagram of a visual editing page provided in another embodiment of the present application;

[0040] Figure 8 A flowchart of a method for processing a visual draft provided in another embodiment of the present application;

[0041] Figure 9 A schematic diagram illustrating the process from design draft to code generation provided in one embodiment of the present application;

[0042] Figure 10 A schematic diagram of the structure of a data processing device provided in one embodiment of the present application;

[0043] Figure 11 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0044] Before describing the solutions provided by the embodiments of the present application, a brief description of the relevant terms involved in the present application is first given.

[0045] Visual image: This refers to the page image corresponding to the visual draft, which is usually generated based on the visual draft file created by designers using design software for a specific application scenario. For example, a visual image can be a page image used to display relevant information about products such as home appliances, clothing, and beauty products.

[0046] Component: The smallest unit of a visual image that cannot be divided further, such as text, icons, pictures, shapes, etc.

[0047] Component: The result of componentizing materials of different granularities on the user interface, such as search boxes, buttons, timers, coupons, video time displays, and other components.

[0048] Container node: A node in the page layout structure that corresponds to a container that can hold and display one or more components.

[0049] Layout information: The hierarchical structure of each component in the visual image in the program code.

[0050] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0051] Some processes described in the specification, claims, and figures of this application include multiple operations that appear in a specific order. These operations may be executed out of the order in which they appear in this document or in parallel. Operation numbers, such as 101 and 102, are merely used to distinguish between different operations and do not imply any specific order of execution. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that terms such as "first" and "second" are used to distinguish between different messages, devices, modules, etc., and do not imply a sequential order or limit the "first" and "second" to different types. The term "or / and" in this application simply describes an association between associated objects, indicating that three possible relationships exist. For example, "A or / and B" indicates that A can exist alone, A and B can exist simultaneously, or B can exist alone. The character " / " in this application generally indicates that the associated objects are in an "or" relationship. Furthermore, the following embodiments represent only some of the embodiments of this application, and are not intended to be exhaustive. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative work shall fall within the scope of protection of this application.

[0052] In the traditional page development process, visual designers need to rely on design tools to design the visual images corresponding to the page. Programmers then manually write code based on the visual images designed by the designers to obtain program codes for restoring the visual images. This manual coding method has low development efficiency and a long development cycle.

[0053] With the rapid development of computer technology, in the existing technology, although there are some methods that can automatically generate page codes, for example, through image recognition technology (such as machine learning models, such as Figure 1 The target detection model obtained through training in the image recognition process) directly recognizes the page image corresponding to the visual draft, and automatically generates program code for restoring the page image corresponding to the visual draft based on the recognition result. However, the program code generated by this image recognition method is basically composed of components with the smallest granularity, such as pictures, texts, etc., which is difficult to meet the componentization requirements of the code development process, and is prone to misrecognition during the recognition process. Figure 1The video component corresponding to the rice cooker position 31 shown in the figure is directly identified as an image element, and the corresponding generated program code uses rax-image to express the code segment corresponding to the rice cooker position 31. However, in the actual development process, the program code segment corresponding to the rice cooker position 11 should be expressed by rax-video. In response to the above problems, the prior art has proposed a method of using deep learning technology to identify the components in the page image corresponding to the visual draft, so as to automatically generate program code for restoring the page image. However, this method is prone to position deviation, wrong recognition, etc. For example, continue to refer to Figure 1 As shown, when the trained target detection model is used to identify the video button component 32 in image 100, the output recognition result may appear as shown in image 110 or image 120, that is, the video button component 32 in image 110 (i.e., 100) or image 120 (i.e., 100) can be accurately identified, but there is a position deviation; or, the output recognition result may also mistakenly identify the text component corresponding to "Enjoy 24 interest-free installments" in image 120 as the video button component 32, and so on. Moreover, when training the target detection model, a large number of samples need to be input, and the sample is the entire image of the visual draft. This requires manual labeling of the components in the sample that the model wants to recognize in order to train a target detection model that can recognize the corresponding components. In addition, when a new component classification is added, each sample needs to be re-labeled, and the labeling cost is relatively high. In summary, it can be seen that the existing automatic code generation method has problems such as low accuracy and high cost, and is difficult to apply to industrial production environments.

[0054] In the process of implementing the technical solution provided by the embodiment of the present application, the inventor discovered that: in addition to the page image corresponding to the visual draft used by the automatic code generation program in the above-mentioned prior art, the visual draft file also contains layer description information of each layer of the page image, such as position information, size information, CSS properties, etc. To this end, the layer description information corresponding to the page image can be extracted from the visual draft file to obtain the layout information of the page image based on the layer description information, and the page image can be cropped according to the layout information to obtain at least one local image corresponding to the component in the page image. After that, by identifying the at least one local image, the program code corresponding to the page image can be obtained according to the identification result of the at least one local image and the layout information. Executing the program code can restore the corresponding page image. In the process of automatically generating the program code corresponding to the page graph, the method of this application directly converts the component identification problem in the page graph into a component classification problem. Since the position information of the component is accurate, as long as the category of the local graph corresponding to the component is accurate, the generated program code will be accurate. Moreover, when training the image classification model for identifying and classifying the local graph, the required samples can be automatically generated by the program, no longer relying on manual labeling, which helps to reduce costs. The specific solution description will be described in detail below.

[0055] Before introducing the method embodiments provided in this application, the hardware architecture on which the technical solutions provided in this application may be based is described.

[0056] In one feasible solution, the method provided in the embodiment of the present application can be based on Figure 2 The system architecture shown in the figure is realized. Figure 2 The data processing system shown in FIG. 1 includes a server 001 and a client 002. The server can be a physical server, a virtual server, a cloud, etc., which is not specifically limited in this embodiment. The client can be any device such as a smart phone, a laptop, a smart wearable device, a desktop computer, etc.

[0057] Client 002, used to send image files to the server;

[0058] The server 001 is used to determine the layout information of the image based on the image file of the image; based on the layout information, the image is processed to determine the local image with component function in the image; based on the layout information and the local image with component function, the first description information for generating program code is obtained.

[0059] In this embodiment, the client 002 and the server 001 may be connected via a wireless or wired network. If the client 002 and the server 001 are connected via a mobile network, the mobile network may be of any type including 4G (LTE), 4G+ (LTE+), WiMax, and 5G.

[0060] In this embodiment, the server 001 can generate the program code corresponding to the image based on the first description information. The client 002 can also receive the program code sent by the server 001 and display the program code for the user to view. Alternatively, the client 002 can request the server 001 to obtain the first description information, and based on the obtained first description information, display a visual layout page related to the image, and the user can participate in the error correction of the first description information through the visual layout page. The error correction data in which the user participates in the error correction can be saved as auxiliary data. These saved auxiliary data can be used to improve the data processing accuracy of the server 001, and more specifically to improve the accuracy of generating the first description information, or to improve the accuracy of program code generation. For example, the user corrects the image recognition result of a local image through the visual layout page. The image recognition result before the error correction and the image recognition result after the error correction can be used as negative samples and positive samples of the local image, respectively, to train the image recognition model and improve the recognition accuracy of the image recognition model.

[0061] The specific workflows of the various components of the data processing system provided in this embodiment, such as the server and the client, and the signaling interactions between them will be further described in the following embodiments and will not be repeated here.

[0062] Currently, the methods provided in the embodiments of the present application can also be completed by a single machine, such as by a client device with certain computing capabilities. In other words, the execution subject of the methods provided in the following embodiments can be a client device.

[0063] Figure 3 The flowchart of the data processing method provided by an embodiment of the present application is shown. The execution subject of the method provided by this embodiment can be the server in the above system embodiment. Figure 1 As shown, the method includes the following steps:

[0064] 101. Determine layout information of the picture according to the picture file;

[0065] 102. Process the image based on the layout information to determine a partial image in the image having a component function;

[0066] 103. Obtain first description information for generating program code based on the layout information and the local diagram having component functions.

[0067] In the above 101, the image file can be a file created by a designer using design software for a specific application scenario and obtained through the corresponding design software. For example, a designer uses the corresponding design software to design a UI page image for an application. After the designer completes the design of the UI page image using the design software, the designer can obtain the image file of the UI page image through the design software. The above-mentioned design software may include, but is not limited to, Sketch, Adobe Illustrator, Photoshop, Adobe XD, and other software.

[0068] The above-mentioned image file usually contains multiple layers used to form an image and layer description information of each layer in the multiple layers, such as position information, size information, CSS properties, etc. Therefore, by parsing the image file, the layer description information of the image can be extracted from the image file to obtain the layout information of the image based on the layer description information. The data format of the layer description information here can be but is not limited to JSON (JavaScript Object Notation) data format and HTML data format. However, in actual situations, there are differences in the focus of designers in the design process and programmers in the programming process. Designers are more concerned about whether the desired visual effect can be achieved in the image file, while programmers are more concerned about the rationality of the layer structure and nesting. For example, designers sometimes add some layers that have no effect on the layout and vision to the image file during the design process; for example, in the image file, designers sometimes use several small layers to splice to achieve the desired visual effect. For example, if you put Figure 1 The rice cooker 321 shown is composed of two graphic elements a1 and a2 located in different layers; but from the perspective of programmers, the graphic formed by the graphic elements a1 and a2 (i.e., the rice cooker 321) can be regarded as a whole. In this case, the layers need to be merged. Based on this, in order to improve the structural rationality and simplicity of the layout information of the picture, before using the layer description information to obtain the layout information of the picture, it is also necessary to perform reprocessing operations such as layer merging and deleting useless layers on the layer description information. That is, in a feasible technical solution, the above step 101 "determining the layout information of the picture based on the picture file" can specifically include:

[0069] 1011. Extracting layer description information of the image from the image file;

[0070] 1012. Perform layer reprocessing on the layer description information to obtain layout information of the image.

[0071] In the above 1011, the image file can be parsed and processed, and then the layer description information of the image can be extracted from the image file. Specifically, the image file data can be automatically read by a corresponding parsing program to extract the layer description information of the image from the image file; wherein, the layer description information may include but is not limited to: position information, size information, CSS (Cascading Style Sheets, Cascading Style Sheets) properties, etc. Of course, the image file can also be parsed using a parsing tool to extract the layer description information of the image from the image file; wherein, the parsing tool can be a software development kit, such as a Software Development Kit, which can be referred to as SDK.

[0072] It should be noted here that the layer description information of the image extracted from the image file is layer description information based on top, bottom, left and right positioning (this positioning is also called absolute positioning). The layout of the image that can be obtained by directly using this layer description information is the layout of this positioning method. Since the layout of this positioning method is not scalable, has poor readability, and is not maintainable for developers, after extracting the layer description information of the image from the image file, the layer description information needs to be further processed such as the layout structure and CSS properties of the layer, so as to obtain layer description information that complies with the layer protocol specification, thereby obtaining layout information that complies with the layer protocol specification. For specific implementation, please refer to the relevant description below.

[0073] The above-mentioned step 1012 of "performing layer reprocessing on the layer description information to obtain the layout information of the image" can be specifically implemented by the following steps:

[0074] A01. Identify the layers contained in the image based on the layer description information;

[0075] A02. Perform layout processing on the layers contained in the image using a layout algorithm to obtain layout information that complies with layer protocol specifications.

[0076] During specific implementation, the layout algorithm can be used to perform at least one of the following layout processing on the layers contained in the image: loop identification, reasonable positioning, positioning mode conversion, redundant nesting deletion, reasonable grouping nesting, etc. In addition, the elements corresponding to the layers contained in the image can also be adaptively processed, for example, the extensibility of the elements themselves, such as the adaptive position and scalability of nodes such as text, images, and components, the alignment relationship between elements, and the maximum width and height tolerance of elements. Specifically, the use of a layout algorithm to perform layout processing on the layers contained in the image or the adaptive processing on the elements corresponding to the layers can be referred to the existing related technologies, and will not be described in detail here. The above-mentioned layer protocol specification can be a D2CUI (i.e., Design2Code User Interface) layer protocol specification, and the layout information obtained that complies with the layer protocol specification can be as follows. Figure 4a A component tree 20 is shown.

[0077] The above method is used to obtain the layout information that complies with the layer protocol specification (such as can be represented as Figure 4a After obtaining the layout information of the component tree shown in FIG, possible components can be cropped out based on the layout information, with the container node as the granularity. That is, in one feasible technical solution, the processing of the image in step 102 may include cropping, image recognition, etc. Specifically, step 102 of "processing the image based on the layout information to determine a partial image in the image that has component functions" may include:

[0078] 1021. Crop the image according to the layout information to obtain at least one partial image.

[0079] 1022. Perform image recognition on at least one partial graph to obtain a partial graph having component functions.

[0080] More specifically, the above step 1021 may include:

[0081] 10211. Determine a cropping area based on the layout information;

[0082] 10212. Crop the image according to the cropping area to obtain a cropped partial image.

[0083] In step 10211 above, since the position information of the container nodes is known in the layout information, the cropping region can be determined based on the position information of the container nodes in the layout information. That is, step 10211, "Determining the cropping region based on the layout information," can be implemented using the following steps:

[0084] A11. Determine a node according to the layout information;

[0085] A12. Get the location information of the node;

[0086] A13. Determine the cropping area based on the position information of the node.

[0087] To facilitate understanding of the above steps, the following will be combined Figure 4a See the example shown. Figure 4a From the layout information of each node in the component tree 20 shown, it is not difficult to see that the nodes in the component tree 20 include two types: container nodes (such as View nodes) and component nodes (such as Text nodes and Image nodes). Among them, the container nodes also have a nested relationship (or parent-child relationship), for example, Figure 4a , the container node View.outer22 is nested in the container node View.bd21; or the container node View.outer22 is called the child node of the container node View.bd21, and the container node View.bd21 is the parent node of the container node View.outer22. In the process of determining the nodes according to the layout information, only the container nodes can be regarded as the nodes that need to be determined, and then the cropping area can be determined according to the position information of the container nodes. For example, based on the position information of View node 21, area 31 can be determined as the cropping area; based on the position information of View node 22, area 32 can be determined as the cropping area. Similarly, other cropping areas in the image 30 can be determined, such as area 33, area 34, area 35, area 36, area 37, and area 38. Of course, both the container node and the component node can be regarded as nodes that need to be determined. At this time, when determining the clipping area according to the position information of the node, in addition to determining the clipping area corresponding to the above-mentioned container node according to the position information of the container node, the clipping area corresponding to the component node can also be determined according to the position information corresponding to the component node. For example, based on the position information of the component node 221, the area 321 can be determined as the clipping area; based on the position information of the component node 211, the area 311 can be determined as the clipping area, and so on. The clipping areas corresponding to other component nodes can be determined (not marked in the figure).

[0088] In the above 10212, according to the cropping area determined in step 10211, the image is cropped to obtain the corresponding partial image. Figure 4aBy cropping the image 30 (i.e., image 10) according to the cropping area determined by the container node, the local graph included in the local graph set 40 can be obtained; and by cropping the image 30 according to the container node and the component node, in addition to obtaining the local graph included in the local graph set 40, the local graph corresponding to the cropping area determined according to the position information of the component node can also be obtained, such as the local graph corresponding to the cropping area 321. It should be noted that: in the layout information, the possible component nodes in the layout information can be determined at the granularity of the container node, and then the local graph obtained by cropping the image according to the cropping area determined by the position information of the container node is the component that may need to be componentized in the image.

[0089] In the above, after obtaining at least one partial image, it is necessary to identify the category to which the image content in the partial image belongs, so as to determine the category to which the partial image belongs (such as a video time display component type, a countdown component type, a scroll bar component type, etc.). Specifically, when determining the category to which the image content in the at least one partial image belongs, an image classification model can be first trained based on a small number of high-quality samples, and then the at least one partial image can be input into the trained image classification model. The trained image classification model is used to extract and analyze features of the at least one partial image, thereby determining the category to which the image content in the at least one partial image belongs.

[0090] That is, the above-mentioned step 1022 of “performing image recognition on at least one partial graph to obtain a partial graph having component functions” may specifically include:

[0091] 10221. Obtain an image classification model;

[0092] 10222. Use the at least one partial image as input to the image classification model, execute the image classification model, and obtain an output result containing the category to which the image content in the at least one partial image belongs;

[0093] 10223. According to the output result, determine that the classification of the graph content is a partial graph of component classification, so as to obtain the partial graph with component function.

[0094] In specific implementation, the image classification model can be obtained based on training of labeled samples, and the labeled samples can be understood as high-quality samples; these samples identified as high-quality can be obtained by professional staff through relevant annotation tools (such as Labelme, Sloth, Vatic) to label the samples; they can also be obtained based on a crowdsourcing task service platform, and then some samples that meet the high-quality requirements are screened out using corresponding evaluation methods. After obtaining the image classification model, the at least one local image can be input into the image classification model, and the image classification model will assign a probability value to each local image. The larger the probability value of a certain classification, the greater the probability that the image classification model predicts that the classification of the content in the local image belongs to that classification. The final classification result can be obtained by comparing the probability value with the preset threshold. For example: set a preset threshold to 0.7. When the probability value of a certain classification is greater than 0.7, it is considered to be the final classification result. For example, Figure 4b After the local images in the local image set 40 obtained are input into the trained image classification model, the probability value of the category to which each local image belongs will be obtained as shown in Table 50. It can be seen from Table 50 that only the probability values of local images 31 and local images 32 are greater than 0.7, so the final classification result is: local images 31 and local images 32 are components of the video time display type (i.e., videobtn components).

[0095] It should be noted here that: after using the image classification model to obtain the category to which the image content in the at least one local image belongs, the classification result corresponding to the at least one local image can also be used as training sample data of the image classification model to improve the accuracy of the image classification model. Of course, in order to ensure the accuracy of the at least one local classification result, the image classification result of a local image can also be corrected so that both the image classification results before and after correction can be used as training sample data of the image classification model to improve the accuracy of the image classification model. The corresponding correction processing operation will be described below. Accordingly, in the process of training the image classification model, it is only necessary to provide a small amount of high-quality sample data at the beginning of the training, and in the subsequent training process, the training sample data can be automatically generated by the program to improve the accuracy of the image classification model.

[0096] In addition, the content that can be supplemented in this embodiment is: this embodiment is aimed at the recognition of component classification, so the corresponding image classification model (such as Figure 4b As shown), the image classification model is used to classify and identify the local image. For example Figure 4bIn the example shown, on an e-commerce application page, component categories generally include videobtn (video time display), countdown (countdown), sliderbar (scrollbar), and others. The image classification model can classify four component categories, so Figure 4b In the example shown, the component category recognition can be completed by selecting the image classification model obtained by training using samples of the above categories. In other application scenarios, when the page component category may not be the above categories, a model different from the above image classification model needs to be used. For this reason, the solution provided in this embodiment can provide a variety of image classification models, which are targeted at different application scenarios. A variety of image classification models can be selected by the user, or they can be automatically selected by the execution subject of the method of this embodiment (such as a server) (such as selecting an image classification model based on task attributes), etc. This embodiment does not specifically limit this.

[0097] In addition to providing a variety of trained and callable image classification models, this embodiment can also provide basic models, such as deep learning models, convolutional neural network models, etc.; and provide users with corresponding interfaces so that users can upload training data, scene-related data, etc. through the client, and then use the training data and scene-related data uploaded by the user to train the basic model selected by the user to obtain a model that meets the scenario required by the user.

[0098] The above-mentioned basic models that users can select and call for training, various image classification models, etc. can all be deployed offline on the server side to provide users with the functions of online application, selection, and use (such as through a web page or a corresponding application APP). In addition to these models, this embodiment also provides network-side users with a service that automatically generates program codes corresponding to images (or page design drawings). When using this service, users can select an image classification model, or they can train the required model by themselves, etc. Different charging plans can be adopted for different service items selected by users; for example, the user chooses to train the required model by himself, and then uses the trained model to complete the automatic generation of program codes corresponding to the image. The charging plan is the sum of the fees corresponding to the code generation service and the training model fees. In specific implementation, corresponding billing strategies can be formulated for each service item provided to the user. In actual billing, billing can be carried out according to the pre-established billing strategy based on the service item category selected by the user, the amount of data processed, etc.

[0099] In this embodiment, the above-mentioned step 103 of “obtaining first description information for generating program code according to the layout information and the partial diagram having component functions” may specifically include:

[0100] 1031. Locate the target node corresponding to the local graph having the component function in the layout information;

[0101] 1032. At the target node, add the content of the local graph corresponding to the target node to obtain second description information; wherein the content of the local graph is determined based on the component classification to which the local graph belongs;

[0102] 1033. Perform conversion processing on the second description information in accordance with code semantics to obtain the first description information.

[0103] In a specific implementation, among the image recognition results of at least one partial graph, some partial graphs may be classified as component, while others may not be classified as component. In this case, the component classification node in the layout information may be determined as the target node to which content is to be added. The component classification node is the node corresponding to the partial graph for which the image recognition result is a component classification.

[0104] In steps 1031 and 1032, the target node can be found from the layout information by calling the corresponding identification function, and then the local graph content corresponding to the target node can be added to the corresponding target node to obtain the second description information. The content of the local graph can be determined based on the component classification to which the local graph belongs, or the content of the local graph is the component classification category to which the graph content in the local graph belongs.

[0105] For example, see Figure 4a and Figure 4b Based on the image recognition results of at least one partial image of the image 10, partial images 31 and 32 are determined to be video time display components (i.e., videobtn components). Based on the image recognition results, a recognition function can be called and executed to find the View 21 node corresponding to partial image 31 and the View 21 node corresponding to partial image 32 from the layout information, and the corresponding image recognition results are added to the Smart attributes corresponding to the View 21 node and the View 21 node, respectively. In the layout information, the View 21 node corresponding to partial image 31 contains a component name parameter (componentName: "View"), a unique identifier name (i.e., id), and a class name (i.e., className), but does not contain a "samrt" attribute. After adding the image recognition results (such as the component classification category), the content of the View 21 node corresponding to partial image 31, in addition to the above information, also contains "smart" related attribute content.

[0106] In actual applications, it may happen Figure 4bAs shown in the situation, there are two partial graphs with the same component classification. For example, the View 21 node and View 22 node corresponding to the partial graph 31 and the partial graph 32 in Figure 10 are both identified as video time display component (i.e., videobtn category) categories, and the View 21 node corresponding to the partial graph 31 and the view 22 node corresponding to the partial graph 32 are nested. For this reason, in the process of semantically transforming the contents corresponding to the View 21 node and the View 22 node corresponding to the partial graph 32 and the partial graph 32, it is necessary to combine the nested relationship between the two to accurately determine the nodes that need to be transformed, thereby semantically transforming the contents corresponding to the nodes that need to be converted. That is, in a feasible technical solution, step 1033 "converting the second description information in accordance with the code semantics to obtain the first description information" can be implemented specifically by the following steps:

[0107] A21. Traverse the second description information and search for at least two nodes in the second description information that are in a nested relationship and belong to the same category;

[0108] A22. If the second description information contains at least two nodes that are in a nested relationship and belong to the same category, determine that the node in the inner layer of the nested relationship among the at least two nodes is the node to be converted;

[0109] A23. Perform semantic conversion on the content corresponding to the node to be converted in the second description information to obtain the first description information.

[0110] In the above steps A21 and A22, some filtering rules may be added to the recognition function. For example, when multiple nodes in a nested containment relationship are identified as the same classification node, only the innermost node is taken as the node to be converted.

[0111] For example, using the recognition results corresponding to partial image 31 and partial image 32 as an example, we can see that there is a nested relationship between the View 21 node corresponding to partial image 31 and the View 22 node corresponding to partial image 32, and both View 21 and View 22 nodes are classified as video time display components (VideoBtn categories). The recognition function code can be configured with relevant filtering rules. Using this recognition function code, it can be determined that the innermost View 22 node is the video time display component node (i.e., the VideoBtn node) that needs to be converted.

[0112] The above-mentioned A23 “performing semantic conversion processing on the content corresponding to the node to be converted in the second description information” includes at least one of the following:

[0113] Change the name parameter in the content corresponding to the node to be converted to the name of the category to which the node to be converted belongs;

[0114] According to the category to which the node to be converted belongs, a corresponding attribute value is added to the content corresponding to the node to be converted.

[0115] For example, following the examples listed in steps A21 and A22 above, after the code program executing the identification function determines that the View 22 node in the second description information is the node that needs to be converted, the expression function can be called for the content corresponding to the View 22 node to replace the name parameter componentName: "View" in the content corresponding to the View 22 node with the category name to which the View 22 node belongs, that is, "VideoBtn". Among them, the expression function is used to process the node that needs to be converted, such as changing the name; the component name is associated according to the component category videobth and the label entered for the component when the component is entered, that is, when entering the component, the component category needs to be entered at the same time for component identification. In addition, the expression function B2 can also extract time information, such as "00:35" as the attribute value of the View 22 node and add it to the content corresponding to the View 22 node.

[0116] The above-mentioned A23 “converting the second description information in accordance with code semantics to obtain the first description information” may further include:

[0117] Determining whether there is a child node under the node to be converted according to the second description information;

[0118] In the case that there are still sub-nodes under the node to be converted, the content corresponding to the sub-nodes under the node to be converted in the second description information is deleted.

[0119] Furthermore, the method provided in this embodiment may further include the following steps:

[0120] 104. Obtain a program code corresponding to the image according to the first description information.

[0121] In a specific implementation, after obtaining the first description information, the first description information can be input into a DSL converter, and the DSL converter generates different types of program codes corresponding to the first description information, such as React, Vue, etc., to restore the image. For example, see Figure 5 As shown, the first description information of the partial image 31 and the partial image 32 of the picture 10 is input into the DSL converter to obtain the program code segments corresponding to the partial image 31 and the partial image 32.

[0122] The method provided in this embodiment determines the layout information of the image based on the image file of the image, and then processes the image based on the layout information to obtain at least one local image, and performs image recognition on the at least one local image to generate first description information based on the layout information and the image recognition result of the at least one local image (such as the component classification result), so as to obtain the program code corresponding to the image according to the first description information. Among them, executing the program code can restore the image. During the entire program code generation process, since the position information of the local image is accurate, as long as the recognition result of the local image is accurate, the generated program code must be accurate; in addition, the scheme of this embodiment is simple, and the accuracy of automatic code generation is improved. When training the model for recognizing the local image, the required samples can be completely automatically generated by the program, and no longer rely on manual labeling, which is conducive to reducing costs.

[0123] Furthermore, the solution provided in this embodiment also provides a user participation interface, which facilitates the user to participate in one or several links of the entire process to improve the accuracy of the overall solution. Specifically, this embodiment may also include the following steps:

[0124] 105a. In response to a visualization display request sent by the client for the first description information, return a visualization arrangement page related to the image generated according to the first description information to the client;

[0125] 105b. Receive operation information on the visual arrangement page sent by the client;

[0126] 105c. Determine, based on the operation information, auxiliary data for improving the accuracy of generating the first description information.

[0127] In the above steps 105a and 105b, the user can send a visualization display request to the execution body of this embodiment for the first description information through the interactive method provided by the client (such as touch, mouse, keyboard). After receiving the visualization request, the execution body of this embodiment will return to the client a visualization arrangement page related to the image generated based on the first description information. The visualization arrangement page provides the user with an operation interface for performing logical arrangement, data arrangement, interactive arrangement, visual arrangement and other operations on the image. Among them, the logical arrangement is to arrange the nested relationship between the components or elements in the image contained in the first description information, the data arrangement is to arrange the position, size, name, component identification classification (i.e., the image recognition result mentioned above) of each component or element in the first description information, etc., and the interactive arrangement and visual arrangement are operations such as adding new components or elements to the image.

[0128] For example, Figure 6 Taking a visual arrangement page shown as an example, through this visual arrangement page, the user can add new components or elements to the image 61, such as adding a button component 610 to the image 61 by dragging and dropping. In addition, the position, size, CSS style, name, etc. of the components or elements of the image can also be modified through the style, property, event, data and other operation modules in the visual arrangement page. Figure 6 As shown, users can also correct the classification recognition results of one or several components (i.e., the image recognition results mentioned above) through the visual arrangement page.

[0129] The above-mentioned step 105c of “determining, based on the operation information, auxiliary data for improving the accuracy of generating the first description information” may specifically include:

[0130] The operation information includes information for correcting an image recognition result of a partial image, and the image recognition result before correction and the image recognition result after correction are both used as training sample data of an image recognition model to improve the accuracy of the image recognition model;

[0131] The image recognition model is used to perform image recognition on the local image.

[0132] For example, see Figure 6 As shown, assuming that the recognition result of the partial image 611 of the picture 61 is an image type, the type of the partial image 611 can be changed by triggering the type conversion 612 control under the attribute list, and the type of the partial image 611 is changed to a video component type, thereby correcting the image recognition result of the partial image 611. After the image recognition result of the partial image 611 is corrected, the image recognition result before and after the correction of the partial image 611 can be used as Figure 4b The training sample data of the image classification model (i.e., image recognition model) is used to improve the accuracy of the image classification model. In this way, in the process of improving the image classification model, the training sample data can be automatically generated by the program, which can reduce the cost of manual labeling.

[0133] The above steps provide a user with an interface for error correction and utilizing the user's error correction to improve the accuracy of the overall solution. In fact, in addition to the error correction interface, this embodiment also provides a user with an interface for directly modifying the first description information. That is, the method provided in this embodiment may also include:

[0134] 106. Modify the first description information according to the operation information, so as to obtain the program code according to the modified first description information.

[0135] Furthermore, the solution provided in this embodiment can also provide a user with an interface for modifying the final program code, that is, the method provided in this embodiment further includes:

[0136] 107a. In response to the request for obtaining the program code sent by the client, return the program code to the client for display on the interface of the client;

[0137] 107b. Receive the modification of the program code fed back by the client;

[0138] 107c. Save the modified program code.

[0139] For example, see Figure 7 As shown, after the user triggers control 613 to obtain the program request corresponding to image 61, the program code E1 of the image 61 will be displayed on the client interface. The user can modify the program code through the interactive method provided by the client, and then save the modified program code by triggering the "Save cmd+s" control 614.

[0140] The method provided in the embodiment of this application will be described below in conjunction with a specific application scenario, namely a design draft designed by a UI designer in a UI interface scenario. Figure 8 FIG. 1 shows a flow chart of a method for processing a visual draft according to another embodiment of the present application. Figure 8 As shown, the visual draft processing method includes:

[0141] 201. Determine layout information of the visual draft according to the visual draft file;

[0142] 202. Based on the layout information, process the page diagram corresponding to the visual draft to determine a partial diagram having component functions in the page diagram;

[0143] 203. Obtain first description information for generating program code according to the layout information and the local diagram having component functions.

[0144] The technical solution provided in this embodiment determines the layout information of the draft visual based on the draft visual file, and then processes the page image corresponding to the draft visual based on the layout information to obtain a partial image with component functions in the page image. The first description information is then generated based on the layout information and the partial image with component functions in the page image. Since the entire process only requires ensuring the accuracy of the recognition result of the partial image, the solution is simple.

[0145] The above 202 "processing the page diagram corresponding to the visual draft based on the layout information to determine the partial diagram with component functions in the page diagram" may specifically include

[0146] 2021. Crop the page image according to the layout information to obtain at least one partial image;

[0147] 2022. Perform image recognition on at least one partial graph to obtain a partial graph having component functions.

[0148] The above-mentioned cropping process of the page image in 2021 may specifically include:

[0149] 20211. Determine a cropping area based on the layout information;

[0150] 20212. Crop the page image according to the cropping area to obtain a cropped partial image.

[0151] The above-mentioned 2022 “performing image recognition on at least one partial graph to obtain a partial graph having component functions” may specifically include:

[0152] 20221. Obtain image classification model;

[0153] 20222. Use the at least one partial image as input to the image classification model, execute the image classification model, and obtain an output result containing the category to which the image content in the at least one partial image belongs;

[0154] 20223. According to the output result, determine that the classification of the graph content is a partial graph of component classification, so as to obtain the partial graph with component function.

[0155] The above-mentioned 203 “obtaining first description information for generating program code according to the layout information and the partial diagram having component functions” may specifically include:

[0156] 2031. Locate the target node corresponding to the local graph having the component function in the layout information;

[0157] 2032. At the target node, add the content of the local graph corresponding to the target node to obtain second description information; wherein the content of the local graph is determined based on the component classification to which the local graph belongs;

[0158] 2033. Perform conversion processing on the second description information in accordance with code semantics to obtain the first description information.

[0159] Furthermore, the method of this embodiment may further include the following steps:

[0160] 204. According to the first description information, obtain the program code corresponding to the page image corresponding to the visual draft.

[0161] Furthermore, the method provided in this embodiment further includes the following steps:

[0162] 205. In response to a visualization display request sent by the client for the first description information, return a visualization arrangement page related to the page map generated according to the first description information to the client;

[0163] 206. Receive operation information on the visual arrangement page sent by the client;

[0164] 207. Determine, based on the operation information, auxiliary data for improving the accuracy of generating the first description information.

[0165] It should be noted here that: for the details of the steps in the data processing method provided in the embodiment of the present application that are not fully described, please refer to the corresponding contents in the above embodiments, and no further details will be given here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include some or all of the other steps in the above embodiments, and the details can be referred to the corresponding contents in the above embodiments. It is only necessary to replace the pictures in the above embodiments with the page pictures of the visual draft in this embodiment.

[0166] More specifically, combined Figure 9 In the embodiment shown, a method for processing a visual draft may include three steps:

[0167] S1. Design to JSON process

[0168] That is, export the layer JSON description information of the image from the UI design draft (i.e. the visual draft mentioned above).

[0169] S2, JSON to JSON process

[0170] Specifically, the image's layer JSON description undergoes layer parsing, layer correction, and layer recognition, and then uses a layout algorithm to obtain a JSON description that conforms to the code structure (i.e., the second description mentioned above). The image is then cropped and its components are identified. The component identification results are then added to the JSON description that conforms to the code structure, and then semantic processing is performed to obtain a JSON description that conforms to the code semantics (i.e., the first description mentioned above).

[0171] S3, JSON to Code Process

[0172] Inputting the JSON description information that conforms to the code semantics obtained in step S2 into the DSL can generate different types of code, such as React, Vue, Rax, H5, MiniApp, etc.

[0173] The above-mentioned step S2JSON to JSON process is the core of the solution of this embodiment, and its specific process may include:

[0174] S21. Obtain the JSON description information of the layer from the design draft through layer parsing;

[0175] S22. The JSON description information of the layer then passes through layer correction and layer recognition before reaching the layout algorithm layer. At this layer, JSON description information that conforms to the code structure is generated, hereinafter referred to as JSON description information with a hierarchical structure.

[0176] S23. After the JSON description information with a hierarchical structure enters the component identification layer, in the organization identification layer, the design draft image is cropped with the container node as the granularity according to the JSON description information with a hierarchical structure to obtain one or more cropped images (i.e., the local images mentioned above); then, an image recognition model (more specifically, an image classification model) is used to predict each cropped image to obtain a cropped image with component function (i.e., the local images mentioned above); finally, the component classification information of the cropped image with component function is appended to the JSON description information with a hierarchical structure to obtain JSON description information with component information.

[0177] S24. The JSON description information with component information obtained in the previous layer is input into the semantic layer. In the semantic layer, the parameters such as the names of the nodes that need to be semantically converted in the JSON description information with component information are modified, and / or corresponding attribute information is added, and / or certain nodes in the JSON description information with component information are deleted, etc., and finally a JSON description information that conforms to the code semantics is obtained.

[0178] An exemplary embodiment of the present application further provides a visual draft processing system. The structure of the visual draft processing system is the same as that of the above-mentioned Figure 2 Specifically, the visual draft processing system includes:

[0179] The client is used to send the visual draft file to the server;

[0180] The server is used to determine the layout information of the visual draft based on the visual draft file; based on the layout information, process the page diagram corresponding to the visual draft to obtain a local diagram with component functions in the page diagram; based on the layout information and the local diagram with component functions, obtain the first description information for generating program code.

[0181] The above-mentioned server can be a physical server, a virtual server, a cloud service platform, etc., and this embodiment does not specifically limit this; the client can be any device such as a smart phone, a laptop, a smart wearable device, a desktop computer, etc.

[0182] The client and server structures in the visual draft processing system can be Figure 2 The corresponding client and server structures are the same. In addition, the execution principles and interaction processes of the various components in the embodiment of the visual draft processing system, such as the client and server, can be found in Figure 8 The description of the corresponding embodiments will not be repeated here.

[0183] Figure 10 Schematic diagram of the structure of a data processing device according to an exemplary embodiment of the present application, the device includes: a first determination module 301, a processing module 302 and an acquisition module 303; wherein,

[0184] A first determining module 301 is configured to determine layout information of a picture according to a picture file of the picture;

[0185] A processing module 302 is configured to process the image based on the layout information to determine a partial image in the image having a component function;

[0186] The obtaining module 303 is configured to obtain first description information for generating program code according to the layout information and the local diagram having component functions.

[0187] The technical solution provided in this embodiment determines the layout information of the image based on the image file of the image, and then processes the image based on the layout information to obtain a partial image with component functions, and then generates first description information based on the layout information and the partial image with component functions. Throughout the process, as long as the recognition accuracy of the partial image with component functions is guaranteed, the generated program code must be accurate. Among them, the recognition of the partial image with component functions can be implemented using a corresponding model, which can be obtained through training. The samples required for training can be automatically generated by the program, and no longer rely on manual labeling, which helps to reduce costs.

[0188] Furthermore, when the above-mentioned processing module 302 processes the image based on the layout information to determine the local image with component function in the image, it is specifically used to: crop the image according to the layout information to obtain at least one local image; perform image recognition on at least one local image to obtain a local image with component function.

[0189] When the processing module 302 crops the image based on the layout information to obtain at least one partial image, it is specifically used to: determine a cropping area according to the layout information; and crop the image according to the cropping area to obtain a cropped partial image.

[0190] Furthermore, when determining the cropping area according to the layout information, the processing model 302 is specifically used to: determine the node according to the layout information; obtain the position information of the node; and determine the cropping area based on the position information of the node.

[0191] Furthermore, when performing image recognition on the at least one local graph, the above-mentioned processing module 302 is specifically used to: obtain an image classification model; use the at least one local graph as input of the image classification model, execute the image classification model, and obtain an output result containing the classification to which the image content in the at least one local graph belongs; and determine, based on the output result, the classification to which the image content belongs is a local graph of component classification, so as to obtain the local graph with component function.

[0192] Furthermore, when the obtaining module 303 obtains the first description information for generating program code according to the layout information and the partial diagram having component functions, it is specifically configured to:

[0193] Locating, in the layout information, a target node corresponding to the local graph having the component function;

[0194] At the target node, adding the content of the local graph corresponding to the target node to obtain second description information; wherein the content of the local graph is determined based on the component classification to which the local graph belongs;

[0195] The second description information is converted in accordance with code semantics to obtain the first description information.

[0196] Furthermore, when the obtaining module 303 performs a conversion process on the second description information in accordance with code semantics to obtain the first description information, it is specifically configured to:

[0197] Traverse the second description information and search for at least two nodes in the second description information that have a nested relationship and belong to the same component category; if the second description information contains at least two nodes that have a nested relationship and belong to the same component category, determine that the node in the inner layer of the nested relationship among the at least two nodes is the node to be converted; perform semantic conversion processing on the content corresponding to the node to be converted in the second description information.

[0198] Furthermore, when the above-mentioned acquisition module 303 performs semantic conversion processing on the content corresponding to the node to be converted in the second description information, it is specifically used to: change the name parameter in the content corresponding to the node to be converted to the component category name to which the node to be converted belongs; and / or add corresponding attribute values to the content corresponding to the node to be converted according to the component category to which the node to be converted belongs.

[0199] Furthermore, when the obtaining module 303 performs a conversion process on the second description information in accordance with code semantics to obtain the first description information, it is further configured to:

[0200] Determining whether there is a child node under the node to be converted according to the second description information;

[0201] In the case that there are still sub-nodes under the node to be converted, the content corresponding to the sub-nodes under the node to be converted in the second description information is deleted.

[0202] Furthermore, when determining the layout information of the image based on the image file of the image, the first determination module 301 is specifically used to: extract the layer description information of the image from the image file; and perform layer reprocessing on the layer description information to obtain the layout information of the image.

[0203] Furthermore, when the first determination module 301 performs layer reprocessing on the layer description information to obtain the layout information of the image, it is specifically used to: identify the layers contained in the image based on the layer description information; and use the layout algorithm to perform layout processing on the layers contained in the image to obtain layout information that complies with the layer protocol specifications.

[0204] Furthermore, the obtaining module 303 is further configured to obtain a program code corresponding to the image according to the first description information.

[0205] Furthermore, the device provided in this embodiment further includes:

[0206] a first response module, configured to respond to a visualization display request sent by a client for first description information and return a visualization arrangement page related to the image generated according to the first description information to the client;

[0207] A first receiving module is configured to receive operation information on the visual arrangement page sent by the client;

[0208] The second determining module is used to determine, based on the operation information, auxiliary data for improving the accuracy of generating the first description information.

[0209] Furthermore, when the above-mentioned second determination module determines the auxiliary data used to improve the accuracy of generating the first description information based on the operation information, it is specifically used: if the operation information includes information for correcting the image recognition result of a local image, then the image recognition result before correction and the image recognition result after correction are both used as training sample data of the image to improve the accuracy of the image recognition model; wherein, the image recognition model is used to perform image recognition on the local image.

[0210] Furthermore, the device provided in this embodiment further includes:

[0211] A modification module is used to modify the first description information according to the operation information, so as to obtain the program code according to the modified first description information.

[0212] Furthermore, the device provided in this embodiment further includes:

[0213] a second response module, configured to return the program code to the client in response to the request for obtaining the program code sent by the client, so as to be displayed on the interface of the client;

[0214] A second receiving module is configured to receive modifications to the program code fed back by the client;

[0215] A saving module is used to save the modified program code.

[0216] It should be noted that the data processing device provided in this embodiment can realize the above Figure 3 The technical solution described in the embodiment of the data processing method shown in FIG. 4 and the specific implementation principles of the above modules or units can be found in the above Figure 3 The corresponding contents in the data processing method embodiment shown are not repeated here.

[0217] Another embodiment of the present application also provides a device for processing a visual draft. The structure of the device for processing the visual draft is the same as that of the above-mentioned Figure 10 Specifically, the visual draft processing device includes: a first determination module, a processing module and an acquisition module;

[0218] A first determining module, configured to determine layout information of the visual draft according to the visual draft file;

[0219] A processing module is used to process the page diagram corresponding to the visual draft based on the layout information to determine a partial diagram having component functions in the page diagram;

[0220] The obtaining module is used to obtain first description information for generating program code according to the layout information and the local diagram with component functions.

[0221] The technical solution provided in this embodiment determines the layout information of the visual draft based on the visual draft file, and then processes the page image corresponding to the visual draft based on the layout information to obtain a partial image with component functions in the page image, and generates first description information based on the layout information and the partial image with component functions in the page image. Since the entire process only requires ensuring that the recognition result of the partial image with component functions is accurate, the solution is simple, and when training the model for recognizing partial images, the required samples can be automatically generated by the program, no longer relying on manual labeling, which helps to reduce costs.

[0222] Furthermore, when the above-mentioned processing module processes the page image corresponding to the visual draft based on the layout information to determine the local image with component function in the page image, it is specifically used to: crop the page image according to the layout information to obtain at least one local image; perform image recognition on at least one local image to obtain a local image with component function.

[0223] Furthermore, when the above-mentioned processing module performs image recognition on at least one local graph to obtain a local graph with component function, it is specifically used to: obtain an image classification model; use the at least one local graph as input of the image classification model, execute the image classification model, and obtain an output result containing the classification to which the image content in the at least one local graph belongs; based on the output result, determine the local graph whose image content belongs to the component classification to obtain the local graph with component function.

[0224] Furthermore, when the above-mentioned acquisition module obtains the first description information for generating program code based on the layout information and the local graph with component function, it is specifically used to: locate the target node corresponding to the local graph with component function in the layout information; at the target node, add the content of the local graph corresponding to the target node to obtain the second description information; wherein, the content of the local graph is determined based on the component classification to which the local graph belongs; and perform conversion processing on the second description information in accordance with the code semantics to obtain the first description information.

[0225] Furthermore, the acquisition module is also used to: obtain the program code corresponding to the page image corresponding to the visual draft based on the first description information.

[0226] Furthermore, the device provided in this embodiment further includes:

[0227] a response module, configured to respond to a visual display request sent by a client for the first description information and return a visual arrangement page related to the page map generated according to the first description information to the client;

[0228] A receiving module, configured to receive operation information on the visual arrangement page sent by the client;

[0229] The second determining module is used to determine, based on the operation information, auxiliary data for improving the accuracy of generating the first description information.

[0230] It should be noted that: the visual draft processing device provided in this embodiment can achieve the above Figure 8 The technical solution described in the embodiment of the method for processing the visual draft shown in the figure, the specific implementation principles of the above modules or units can be found in the above Figure 8 The corresponding contents in the data processing method embodiment shown are not repeated here.

[0231] Figure 11 FIG. 1 shows a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. Figure 11 As shown, the client device includes: a memory 601 and a processor 602. The memory 601 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory 601 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0232] The processor 602 is coupled to the memory 601 and is configured to execute the program stored in the memory 601 to:

[0233] Determining layout information of the picture according to the picture file;

[0234] Processing the image based on the layout information to determine a partial image in the image having a component function;

[0235] First description information for generating program code is obtained according to the layout information and the local diagram having component functions.

[0236] In particular, when the processor 602 executes the program in the memory 601 , in addition to the above functions, it can also implement other functions, for which details please refer to the description of the previous embodiments.

[0237] Further, if Figure 11 As shown, the electronic device further includes: a communication component 603, a power supply component 604, a display 605 and other components. Figure 11Only some components are shown schematically, which does not mean that the electronic device only includes Figure 11 Components shown.

[0238] An embodiment of the present application also provides another electronic device, the structure of which is the same as the above Figure 10 Similarly. Specifically, the electronic device includes: a memory and a processor. The memory is used to store programs. The processor, coupled to the memory, is used to execute the program stored in the memory to:

[0239] Determining layout information of the visual draft according to the visual draft file;

[0240] Based on the layout information, the page diagram corresponding to the visual draft is processed to determine a partial diagram having component functions in the page diagram;

[0241] First description information for generating program code is obtained according to the layout information and the local diagram having component functions.

[0242] In addition to the above functions, the processor can also implement other functions when executing the program in the memory. For details, please refer to the description of the previous embodiments.

[0243] Further, if Figure 11 As shown, the electronic device also includes: communication components, power components, display components and other components. Figure 11 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 10 Components shown.

[0244] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the data processing method steps or functions provided in the above embodiments.

[0245] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0246] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0247] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Determining layout information of the picture according to the picture file; Processing the image based on the layout information to determine a partial image in the image having a component function; Locating, in the layout information, a target node corresponding to the local graph having the component function; At the target node, adding the content of the local graph corresponding to the target node to obtain second description information; wherein the content of the local graph is determined based on the component classification to which the local graph belongs; The second description information is converted to conform to code semantics to obtain first description information used to generate program code.

2. The method according to claim 1, characterized in that Processing the image based on the layout information to determine a partial image in the image having a component function includes: Performing cropping on the image according to the layout information to obtain at least one partial image; Image recognition is performed on at least one partial image to obtain a partial image having component functions.

3. The method according to claim 2, characterized in that Based on the layout information, the image is cropped to obtain at least one partial image, including: Determining a cropping area according to the layout information; The image is cropped according to the cropping area to obtain a cropped partial image.

4. The method according to claim 3, characterized in that Determining a cropping area according to the layout information includes: Determining a node according to the layout information; Get the location information of the node; The cropping area is determined based on the position information of the node.

5. The method according to claim 2, characterized in that Performing image recognition on the at least one partial graph to obtain a partial graph having a component function, comprising: Get the image classification model; Using the at least one partial image as input to the image classification model, executing the image classification model to obtain an output result containing the classification to which the image content in the at least one partial image belongs; According to the output result, it is determined that the classification to which the graph content belongs is a partial graph of component classification, so as to obtain the partial graph with component function.

6. The method according to any one of claims 1 to 5, characterized in that The second description information is converted to conform to code semantics to obtain first description information for generating program code, including: Traversing the second description information, searching for at least two nodes in the second description information that are in a nested relationship and belong to the same component classification; If the second description information contains at least two nodes that are in a nested relationship and belong to the same component category, determining that the node in the inner layer of the nested relationship among the at least two nodes is the node to be converted; Perform semantic conversion processing on the content corresponding to the node to be converted in the second description information.

7. The method according to claim 6, characterized in that Performing semantic conversion processing on the content corresponding to the node to be converted in the second description information includes at least one of the following: Change the name parameter in the content corresponding to the node to be converted to the name of the component category to which the node to be converted belongs; According to the component classification to which the node to be converted belongs, a corresponding attribute value is added to the content corresponding to the node to be converted.

8. The method according to claim 6, characterized in that Performing a conversion process on the second description information in accordance with code semantics to obtain first description information for generating program code further includes: Determining whether there is a child node under the node to be converted according to the second description information; In the case that there are still sub-nodes under the node to be converted, the content corresponding to the sub-nodes under the node to be converted in the second description information is deleted.

9. The method according to any one of claims 1 to 5, characterized in that Determining layout information of the image according to the image file includes: Extracting layer description information of the image from the image file; The layer description information is subjected to layer reprocessing to obtain layout information of the image.

10. The method according to claim 9, characterized in that Performing layer reprocessing on the layer description information to obtain layout information of the image, including: Based on the layer description information, identifying the layers contained in the image; The layers contained in the image are subjected to layout processing by using a layout algorithm to obtain layout information that complies with the layer protocol specification.

11. The method according to any one of claims 1 to 5, characterized in that Also includes: According to the first description information, a program code corresponding to the image is obtained.

12. The method according to any one of claims 1 to 5, characterized in that Also includes: In response to a visualization display request sent by the client for the first description information, returning a visualization arrangement page related to the image generated according to the first description information to the client; Receiving operation information on the visual arrangement page sent by the client; Auxiliary data for improving the generation accuracy of the first description information is determined according to the operation information.

13. The method according to claim 12, characterized in that Determining, according to the operation information, auxiliary data for improving the accuracy of generating the first description information includes: The operation information includes information for correcting an image recognition result of a partial image, and the image recognition result before correction and the image recognition result after correction are both used as training sample data of an image recognition model to improve the accuracy of the image recognition model; The image recognition model is used to perform image recognition on the local image.

14. The method according to claim 12, characterized in that Also includes: The first description information is modified according to the operation information, so as to obtain the program code according to the modified first description information.

15. The method according to claim 13, characterized in that Also includes: In response to a request for obtaining the program code sent by the client, returning the program code to the client for display on an interface of the client; receiving modifications to the program code fed back by the client; The modified program code is saved.

16. A data processing system, characterized in that: include: The client is used to send image files to the server; The server is configured to obtain the layout information of the picture according to the picture file of the picture; Based on the layout information, the image is processed to determine a local graph with component functions in the image; in the layout information, a target node corresponding to the local graph with component functions is located; at the target node, the content of the local graph corresponding to the target node is added to obtain second description information; wherein the content of the local graph is determined based on the component classification to which the local graph belongs; the second description information is converted in accordance with code semantics to obtain first description information for generating program code.

17. A method for processing a visual draft, characterized in that: include: Determining layout information of the visual draft according to the visual draft file; Based on the layout information, the page diagram corresponding to the visual draft is processed to determine a partial diagram having component functions in the page diagram; Locating, in the layout information, a target node corresponding to the local graph having the component function; At the target node, adding the content of the local graph corresponding to the target node to obtain second description information; wherein the content of the local graph is determined based on the component classification to which the local graph belongs; The second description information is converted to conform to code semantics to obtain first description information used to generate program code.

18. The method according to claim 17, characterized in that Based on the layout information, the page diagram corresponding to the visual draft is processed to determine a partial diagram having component functions in the page diagram, including: Performing cropping processing on the page image according to the layout information to obtain at least one partial image; Image recognition is performed on at least one partial image to obtain a partial image having component functions.

19. The method according to claim 18, characterized in that Performing image recognition on at least one partial graph to obtain a partial graph having a component function, including: Get the image classification model; Using the at least one partial image as input to the image classification model, executing the image classification model to obtain an output result containing the classification to which the image content in the at least one partial image belongs; According to the output result, it is determined that the classification to which the graph content belongs is a partial graph of component classification, so as to obtain the partial graph with component function.

20. The method according to any one of claims 17 to 19, characterized in that Also includes: According to the first description information, the program code corresponding to the page image corresponding to the visual draft is obtained.

21. The method according to any one of claims 17 to 19, characterized in that Also includes: In response to a visualization display request sent by the client for the first description information, returning a visualization arrangement page related to the page map generated according to the first description information to the client; Receiving operation information on the visual arrangement page sent by the client; Auxiliary data for improving the generation accuracy of the first description information is determined according to the operation information.

22. A visual draft processing system, characterized in that: include: The client is used to send the visual draft file to the server; The server is configured to determine layout information of the visual draft according to the visual draft file; Based on the layout information, the page diagram corresponding to the visual draft is processed to determine the local diagram with component functions in the page diagram; in the layout information, the target node corresponding to the local diagram with component functions is located; at the target node, the content of the local diagram corresponding to the target node is added to obtain second description information; wherein, the content of the local diagram is determined based on the component classification to which the local diagram belongs; the second description information is converted in accordance with the code semantics to obtain first description information for generating program code.

23. An electronic device, characterized in that: include: memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Determining layout information of the picture according to the picture file; Processing the image based on the layout information to determine a partial image in the image having a component function; Locating, in the layout information, a target node corresponding to the local graph having the component function; At the target node, adding the content of the local graph corresponding to the target node to obtain second description information; wherein the content of the local graph is determined based on the component classification to which the local graph belongs; The second description information is converted to conform to code semantics to obtain first description information used to generate program code.

24. An electronic device, characterized in that: include: memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Determining layout information of the visual draft according to the visual draft file; Based on the layout information, the page diagram corresponding to the visual draft is processed to determine a partial diagram having component functions in the page diagram; Locating, in the layout information, a target node corresponding to the local graph having the component function; At the target node, adding the content of the local graph corresponding to the target node to obtain second description information; wherein the content of the local graph is determined based on the component classification to which the local graph belongs; The second description information is converted to conform to code semantics to obtain first description information used to generate program code.

Images