Layer processing method, model generation method and device

Abstract

The embodiment of the present application provides a layer processing method, a model generation method and a device. The method includes: obtaining a target image corresponding to an interface design draft; determining at least one target area in the target image that meets the merging requirements; based on the intersection information between the at least one target area and different layers, identifying multiple target layers corresponding to the at least one target area, wherein the multiple target layers are used for merging processing as a component corresponding to the target area. The technical solution provided by the embodiment of the present application improves the efficiency and accuracy of layer search.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a layer processing method, a model generation method and a device. Background Art

[0002] The interface design draft is a prototype for implementing graphical interfaces such as user interfaces (UI). One of the main tasks of front-end development is to implement front-end code through interface design drafts. In order to improve development efficiency, some studies have proposed using intelligent algorithms to automatically generate front-end code based on interface design drafts to improve code generation efficiency.

[0003] However, there are some special layers in the interface design draft, such as fragmented layers, etc. These fragmented layers will lead to low quality of generated code. The quality of code generation can be improved by merging the fragmented layers.

[0004] Therefore, how to efficiently and accurately determine the layers that need to be merged in the interface design draft becomes a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0005] The embodiments of the present application provide a layer processing method, a model generation method and a device to solve the technical problem that the prior art cannot efficiently and accurately find layers.

[0006] In a first aspect, an embodiment of the present application provides a layer processing method, comprising:

[0007] Get the target image corresponding to the interface design draft;

[0008] Determine at least one target area in the target image that meets the merging requirements;

[0009] Based on the intersection information between the at least one target area and different layers, a plurality of target layers respectively corresponding to the at least one target area are identified, wherein the plurality of target layers are used for merging processing to serve as a component corresponding to the target area.

[0010] In a second aspect, an embodiment of the present application provides a model generation method, comprising:

[0011] Acquire a first sample image and spatial information of at least one sample area marked by the first sample image and meeting the merging requirements; the at least one sample area corresponds to a plurality of target layers respectively;

[0012] Using the first sample image as input data and the spatial information of the at least one sample area as a training label, training a detection model;

[0013] Among them, the detection model is used to determine at least one target area in the target image that meets the merging requirements based on the target image corresponding to the interface design draft; the at least one target area is used to identify multiple target layers corresponding to each other based on intersection information with different layers respectively; the multiple target layers are used for merging processing to serve as a component corresponding to the target area.

[0014] In a third aspect, an embodiment of the present application provides a layer processing device, including:

[0015] The acquisition module is used to obtain the target image corresponding to the interface design draft;

[0016] A determination module, used to determine at least one target area in the target image that meets the merging requirements;

[0017] An identification module is used to identify multiple target layers corresponding to the at least one target area based on the intersection information between the at least one target area and different layers, wherein the multiple target layers are used for merging processing to serve as a component corresponding to the target area.

[0018] In a fourth aspect, an embodiment of the present application provides a model generation device, comprising:

[0019] An acquisition module, used to acquire the first sample image and spatial information of at least one sample area marked by the first sample image and meeting the merging requirements; the at least one sample area corresponds to a plurality of target layers respectively;

[0020] A training module, configured to train a detection model by using the first sample image as input data and the spatial information of the at least one sample area as a training label;

[0021] Among them, the detection model is used to determine at least one target area in the target image that meets the merging requirements based on the target image corresponding to the interface design draft; the at least one target area is used to identify multiple target layers corresponding to each other based on intersection information with different layers respectively; the multiple target layers are used for merging processing to serve as a component corresponding to the target area.

[0022] In a fifth aspect, a computing device is provided in an embodiment of the present application, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called by the processing component and execute the layer processing method described in the first aspect or the model generation method described in the second aspect.

[0023] In a sixth aspect, a computer storage medium is provided in an embodiment of the present application, storing a computer program, which, when executed by a computer, implements the layer processing method described in the first aspect or the model generation method described in the second aspect.

[0024] In an embodiment of the present application, for a target image obtained by converting an interface design draft, at least one target area in the target image that meets the merging requirements is determined; based on the intersection information of at least one target area and different layers, multiple target layers corresponding to the at least one target area are identified; the multiple target layers corresponding to the target area are used for merging processing as a component corresponding to the target area. Based on visual information, an embodiment of the present application detects a target area in an interface design draft that needs to be merged, and obtains multiple target layers corresponding to the target area based on the intersection information, thereby improving the efficiency and accuracy of layer search.

[0025] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A system architecture diagram is shown in which the technical solution of the embodiment of the present application is applied;

[0028] Figure 2 A flowchart of an embodiment of a layer processing method provided by an embodiment of the present application is shown;

[0029] Figure 3 A flow chart of an embodiment of a model generation method provided in an embodiment of the present application is shown;

[0030] Figure 4 A flowchart of another embodiment of a layer processing method provided by an embodiment of the present application is shown;

[0031] Figure 5 A schematic diagram of scene interaction in a practical application of the technical solution of an embodiment of the present application is shown;

[0032] Figure 6 A schematic diagram of a layer processing process in a practical application of the technical solution proposed in an embodiment of the present application is shown;

[0033] Figure 7A schematic structural diagram of an embodiment of a layer processing device provided by the present application is shown;

[0034] Figure 8 A schematic diagram showing the structure of an embodiment of a model generation device provided by the present application is shown;

[0035] Fig. 9 A schematic diagram of the structure of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0037] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0038] The technical solution of the embodiment of the present application can be applied to application scenarios where there is a need for layer merging, such as in scenarios where front-end code is intelligently developed based on an interface design draft.

[0039] Taking the front-end code development scenario of the user interface (UI) as an example, in the actual generation design process of designers, in order to achieve the desired visual effect, designers usually use multiple fragmented layers to express a UI component, which can be a component in the user interface that can independently express visual effects, such as icons, atmosphere maps, background maps, or controls that can perceive user operations, etc. In the process of implementing this application, the inventor found that this design method of the interface design draft will lead to code redundancy and affect the quality of the final generated code. In order to improve the instructions for generating code, it is necessary to adopt higher design standards to design the interface design draft. However, this will inevitably increase the design cost of designers. Therefore, the fragmented layers expressing the same component can be merged and processed, and they can be coded as a component during code development, so as to improve the quality of code generation.

[0040] So how to find the fragmented layers that express the same component becomes a technical problem that needs to be solved. The inventor thought that a manual assistance method could be used to find multiple layers that need to be merged from the interface design draft and then merge them. However, this method requires manual judgment and screening, which is costly and inaccurate.

[0041] Therefore, in order to improve the efficiency and accuracy of layer search, the inventor has proposed the technical solution of the present application after a series of studies. In an embodiment of the present application, a target image corresponding to the interface design draft is obtained; at least one target area in the target image that meets the merging requirements is determined; based on the intersection information of the at least one target area and different layers, multiple target layers corresponding to the at least one target area are identified, wherein the multiple target layers are used for merging processing as a component corresponding to the target area. The technical solution provided by the embodiment of the present application is based on visual information. The detection model can be used to detect the target area that needs to be merged in the interface design draft, and multiple target layers corresponding to the target area can be obtained according to the intersection information, thereby improving the efficiency and accuracy of layer search. The multiple target layers corresponding to the target area can be merged and converted into the same component for processing during the intelligent code development process, thereby improving the quality of code generation.

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0043] Figure 1 A system architecture diagram that can be applied to the technical solution of an embodiment of the present application is shown, and the system architecture may include a design end 101 and a processing end 102.

[0044] The design terminal 101 can provide a design interface and generate a corresponding interface design draft according to the design operation of the designer.

[0045] The design end 101 and the processing end 102 may be connected via a network. The network provides a medium for a communication link between the design end 101 and the processing end 102. The network may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0046] The processing end 102 can obtain the interface design draft from the design end 101, and detect the existence of at least one target area that meets the merging requirements and multiple target layers corresponding to the at least one target area according to the technical solution of the embodiment of the present application. The multiple target layers corresponding to each target area can be merged.

[0047] Of course, in other implementations of the present application, there is no need to establish a network connection between the design end 101 and the processing end 102. The interface design draft of the design end 101 can be transmitted to the processing end 102 in other ways, such as being transferred from the design end 101 to the processing end 102 via a mobile storage device.

[0048] The design end 101 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language 5th Edition) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application. The design end 101 is usually deployed in an electronic device and needs to rely on the device to run or some apps in the device to run. For ease of understanding, Figure 1 The design end is shown in the form of an electronic device. The electronic device may have a display screen and support information browsing, such as a smart phone, a mobile computer, a desktop computer, or a tablet. The electronic device may also be installed with various other types of applications, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, mailbox design end, social platform software, etc.

[0049] The design end 101 may generally refer to one of a plurality of design ends. This embodiment is only illustrated by taking the design end 101 as an example. Those skilled in the art will appreciate that the number of the above-mentioned design ends may be more or less. For example, there may be only a few of the above-mentioned design ends, or there may be dozens or hundreds of the above-mentioned design ends, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the design ends.

[0050] The processing end 102 may include servers that provide various services, such as a server for background training that provides support for detection models, or a server that performs layer processing on interface design drafts.

[0051] The processing end 102 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0052] In addition, the system architecture may further include a code generation terminal 103 , and the interface design draft processed by the processing terminal 102 may be transmitted to the code generation terminal 103 , and the code generation terminal 103 may encode and generate corresponding target codes.

[0053] Of course, in other implementations of the present application, the processing end 102 may also generate corresponding target codes based on the interface design draft.

[0054] The code generation end 103 may include a server that provides various services, such as a server that provides intelligent code generation.

[0055] It should be noted that the layer processing method and model generation method provided in the embodiment of the present application are executed by the processing end 102, etc.

[0056] It should be understood that Figure 1 The number of each end is only for illustration and can be any number according to actual needs.

[0057] The implementation details of the technical solution of the embodiment of the present application are elaborated in detail below.

[0058] Figure 2 A flowchart of an embodiment of a layer processing method provided in an embodiment of the present application, which can be applied to a processing end, may include the following steps:

[0059] 201: Get the target image corresponding to the interface design draft.

[0060] The interface design draft may be generated by the design end based on the design operation of the designer. The design end may be, for example, a vector drawing application software (Sketch), or other types of drawing tools, such as Adobe Photoshop. The interface design draft may correspond to at least one drawing board, each of which may be used to draw a page in the interface, etc. The drawing board information of at least one drawing board included in the interface design may be converted into a corresponding interface image. For example, the processing end may call the interface provided by Sketch to convert the interface design draft into a corresponding interface image.

[0061] The target image can be any interface image or any segmented image obtained by cutting any interface image, etc., which will be introduced in detail below.

[0062] Of course, the interface design draft may also be composed of graphic files, and the target image may refer to any graphic file of the interface design draft.

[0063] 202: Determine at least one target region in the target image that meets the merging requirement.

[0064] Each target region may correspond to a component, and the component is obtained by editing multiple target layers. The target region may refer to the smallest region in the target image that includes the multiple target layers.

[0065] Extracting at least one target region that meets the merging requirements from the target image can be implemented in a variety of ways, for example, it can be obtained by matching from the target image based on the image features of the region to be merged, etc. Of course, it can also be obtained by using detection model recognition.

[0066] As a possible implementation, extracting at least one target region that meets the merging requirement from the target image may be implemented as follows: based on the pre-configured image features of the region to be merged, extracting the target region that matches the image features of the region to be merged from the target image. The merging requirement may include the pre-configured image features of the region to be merged.

[0067] As another possible implementation manner, extracting at least one target region that meets the merging requirement from the target image may be implemented as follows: using a detection model to determine at least one target region that meets the merging requirement in the target image.

[0068] The detection model may be pre-trained using the spatial information of the sample image and at least one sample area annotated in the sample image that meets the merging requirements.

[0069] That is, the detection model can also be used to extract the target area that meets the merging requirements from the target image.

[0070] In some embodiments, the method may further include:

[0071] Determine a color block image generated by color filling of multiple layers corresponding to the target image; wherein different layers are filled with different colors;

[0072] The determining of at least one target area in the target image that meets the merging requirement may include:

[0073] Image features are extracted from the target image and the color block image of the target image, and at least one target area in the target image that meets the merging requirements is identified based on the image features.

[0074] There are multiple possible implementation methods for extracting image features from the target image and the color block image of the target image: as a possible implementation method, the color block image and the target image may be first fused to obtain a fused image, and then the image features may be extracted from the fused image;

[0075] As another possible implementation manner, the first feature may be extracted from the target image and the second feature may be extracted from the color block image of the target image, and the first feature and the second feature may be fused to obtain the image feature.

[0076] The first feature and the second feature may be fused by cumulative summation or weighted summation to obtain the image feature.

[0077] Among them, feature extraction can be achieved using a pre-trained feature extraction model or image recognition algorithm, etc., of course, it can also be achieved by a detection model, etc. This application does not impose any specific restrictions on this.

[0078] The image recognition algorithm may be, for example, a Histogram of Oriented Gradient (HOG), a Local Binary Pattern (LBP), or a Haar-like feature algorithm, etc. The feature extraction model may be a deep residual network (such as Resnet-50) or a Feature Pyramid Network (FPN), or a combination of the two.

[0079] 203: Based on the intersection information between the at least one target area and different layers, identify a plurality of target layers corresponding to the at least one target area.

[0080] The intersection information is used to indicate the degree of intersection between at least one target area and different layers.

[0081] The intersection information may be information such as the intersection area and / or the layer spacing, etc. The layer spacing refers to the distance between layers.

[0082] Based on the intersection information between at least one target area and different layers, multiple target layers whose intersection information respectively corresponds to at least one target area meets a merging condition may be identified. The merging condition may include, for example, that the intersection area is greater than an area threshold, and / or that the layer spacing meets a distance requirement.

[0083] It can be understood that, from the above description, the target area is at least the minimum area containing multiple target layers. Therefore, the multiple layers corresponding to the target area can be determined based on the intersection area between the target area and different layers. Since the target area corresponding to some components is relatively large, there may be layers that do not belong to the target area in the multiple layers corresponding to the target area determined based on the intersection area between the target area and different layers. For example, the area of ​​the target area corresponding to the background component is relatively large, and the multiple layers corresponding to it may include a text layer, which does not belong to the background component. Therefore, the text layer needs to be filtered out. Since the distances between multiple layers belonging to the same component are all within the preset threshold range, the multiple target layers corresponding to the target area can be determined based on the distances between the multiple layers corresponding to the target area.

[0084] Among them, each target area corresponds to multiple target layers, which can be used as a group of layer groups for merging processing.

[0085] In this embodiment, for the target image corresponding to the interface design draft, at least one target area in the target image that meets the merging requirements can be determined; then, based on the intersection information of at least one target area and different layers, multiple target layers corresponding to at least one target area are identified; wherein the multiple target layers corresponding to each area are used for merging processing. The embodiment of the present application realizes the merging of layers that need to be merged, improves the efficiency and accuracy of layer search, and thus can also improve the efficiency of layer merging processing. Multiple target layers corresponding to the target area can be merged and processed during the intelligent code development process and converted into the same component for processing, thereby improving the quality of code generation.

[0086] The detection model can be used to automatically locate and obtain at least one target area that needs to be merged based on the visual information of the interface design draft. In order to describe the generation process of the detection model in detail. Figure 3 FIG. 1 is a flow chart of an embodiment of a model generation method provided in an embodiment of the present application. The method may include the following steps:

[0087] 301: Obtain a first sample image and spatial information of at least one sample area marked in the first sample image that meets the merging requirement.

[0088] Among them, at least one sample area corresponds to multiple target layers, and can be an area selected by the user that meets the merging requirements.

[0089] The target layer may be, for example, a fragmented layer, that is, a layer that needs to be merged.

[0090] There are multiple ways to obtain the first sample image. As an optional way, the first sample image can be an interface image converted from any artboard in the sample design draft. Therefore, the method can also include: converting at least one artboard in the sample design draft to generate a corresponding interface image; and using any interface image as the first sample image.

[0091] Among them, the sample design draft can be composed of multiple artboards, so the sample design draft can be converted into multiple interface pictures.

[0092] As another optional method, in order to speed up the model training, improve the model accuracy, etc., the first sample image can also be a segmented image obtained by segmenting the interface image to obtain a segmented image of consistent size. Therefore, the method can also include: converting at least one drawing board in the sample design draft to generate a corresponding interface image; segmenting any interface image according to the segmentation requirements to obtain at least one segmented image, and using any segmented image as the first sample image.

[0093] The segmentation requirement may refer to segmentation based on the length of the short side, for example; it may also include segmentation size, etc., to ensure that the components will not be segmented during the segmentation process.

[0094] In addition, in order to ensure that each sample is a valid sample, interface images that do not meet the image requirements are filtered out from at least one interface image corresponding to the sample design draft.

[0095] For example, filter out interface images that do not include the target layer, images that are too small or too large, etc.

[0096] In addition, after each interface image is segmented, there may be images with similar contents. Therefore, the method may further include: deleting any segmented image from two segmented images that meet the similarity requirement according to the image similarity.

[0097] The similarity requirement may be that the similarity exceeds a similarity threshold, wherein the similarity threshold may be flexibly set according to specific circumstances.

[0098] The spatial information may include location information, size information, and the like.

[0099] Each sample region may be manually marked in advance in the first sample image, and spatial information of each sample region may be determined based on the marked position.

[0100] In addition, at least one layer group that needs to be merged can be manually screened from the multiple layers corresponding to the first sample image, each layer group includes multiple target layers, and a merge identifier is set for each layer group, and the multiple target layers included in each layer group correspond to a sample area in the first sample image. Therefore, as another optional method, the method may also include: determining at least one layer group with a merge identifier set in the multiple layers corresponding to the first sample image; at least one layer group includes multiple target layers respectively; and using the spatial information corresponding to any layer group as the spatial information of a sample area that meets the merge requirements. That is, the layer groups can be pre-marked in the sample design draft, and merge identifiers can be set for them. Each layer group consists of multiple target layers that express the same component. The spatial information corresponding to the layer component can be used as the spatial information of the sample area.

[0101] 302: Train a detection model by using the first sample image as input data and spatial information of at least one sample region as a training label.

[0102] Among them, the detection model is used to determine at least one target area in the target image that meets the merging requirements based on the target image corresponding to the interface design draft; at least one target area is used to identify multiple target layers corresponding to each other based on the intersection information with different layers respectively; and multiple target layers are used for merging processing as a component corresponding to the target area.

[0103] In this embodiment, by obtaining the spatial information of the first sample image and at least one sample area marked by the first sample image that meets the merging requirements, the first sample image is used as input data and the spatial information of at least one sample area is used as a training label to train the detection model, so that the trained detection model can be obtained, so as to use the detection model to determine at least one target area in the target image that meets the merging requirements, and then based on the intersection information of at least one target area with different layers, multiple target layers corresponding to each target area can be determined, thereby improving the efficiency and accuracy of target layer search.

[0104] In some embodiments, in order to enrich the number of samples, the method may further include: deleting a predetermined proportion of layers from multiple layers that do not have a merge flag set in any interface image to obtain an extended image; and using the extended image as the first sample image. Alternatively, the extended image is segmented according to the segmentation requirements to obtain at least one segmented image, and any segmented image is used as the first sample image.

[0105] Among them, the layers without a merge flag are layers that do not need to be merged. Therefore, deleting a certain proportion of layers without a merge flag will not affect the result of layer processing of the image, and can generate extended images, enriching the number of samples. Segmenting the sample image according to the segmentation requirements to obtain at least one segmented image can not only enrich the number of samples, but also effectively improve the efficiency of model training by segmenting some sample images that occupy a large amount of processing power.

[0106] In some embodiments, deleting a predetermined proportion of layers from multiple layers in any interface image that do not have a merge identifier set to obtain an extended image may include: deleting a predetermined proportion of layers that meet a deletion condition from multiple layers in any interface image that do not have a merge identifier set to obtain an extended image.

[0107] Among them, when the target layer is a fragmented layer, the deletion condition may be, for example, that the layer width is less than a first threshold or the layer area is greater than a second threshold, wherein the first threshold and the second threshold may be flexibly set according to actual conditions.

[0108] In practical applications, in order to facilitate data processing, the sample design draft can be parsed into a file format with a hierarchical structure, such as a JSON (JavaScript Object Notation, a lightweight data exchange format) file. The JSON file is traversed to obtain multiple layers without the hierarchical structure. Then, the layers are deleted from the multiple layers in any interface image that do not have a merge identifier. After deleting the layers, the sample design draft is updated, and the updated sample design draft is converted into interface images. These interface images can be used as extended images, realizing the generation of new training data based on the original data, thereby achieving the effect of data enhancement.

[0109] Among them, in each training round, a predetermined proportion of layers can be deleted from multiple layers without merge marks in any interface image to obtain an extended image, so that training samples can be randomly generated in each training round, thereby improving the diversity of training samples and further improving the generalization ability of the model.

[0110] In some embodiments, the detection model may include a feature extraction network and a region detection network.

[0111] The first sample image is used as input data and the spatial information of at least one sample area is used as a training label. The training detection model includes: inputting the first sample image into a feature extraction network to extract the first sample image feature; using the first sample image feature as input data of a region detection network and the spatial information of at least one sample area as a training label to train the region detection network.

[0112] Among them, the feature extraction network can be obtained in advance using the second sample image and the corresponding second sample image features, the second sample image is used as input data of the feature extraction network, and the second sample image features are used as training labels to train the feature extraction network.

[0113] Among them, the feature extraction network can be implemented in many ways. For example, it can be composed of a deep residual network (such as Resnet-50) and a feature pyramid network (English: Feature Pyramid Network, abbreviated as: FPN), etc. Of course, the present application is not limited to this.

[0114] In some embodiments, in order to further improve the accuracy of the model, the first sample image and the color block image corresponding to the first sample image may be input into a feature extraction network to extract features of the first sample image.

[0115] As an optional method, a feature extraction network can be used to extract a first sample feature from a first sample image, and a second sample feature can be extracted from a color block image corresponding to the first sample image; the first sample feature and the second sample feature are then fused to obtain the first sample image feature.

[0116] The fusion processing of the first sample feature and the second sample feature may be superimposing and summing the first sample feature and the second sample feature or weighted summing the first sample feature and the second sample feature, for example, superimposing and summing the first sample feature and the second sample feature or weighted summing the first sample feature and the second sample feature in a channel dimension.

[0117] As another optional method, in order to further improve the accuracy of model detection, the first sample image and its corresponding color block image may be first fused to obtain a fused sample image, and then the first sample image features may be extracted from the fused sample image using a feature extraction network.

[0118] The fusion processing of the first sample image and its corresponding color block image can be performed based on the positions of corresponding layers in the first sample image and its corresponding color block image. It can be understood that the color block image is obtained from the first sample image. Therefore, the first sample image and its corresponding color block image include multiple completely consistent layers. The same layers in the first sample image and its corresponding color block image are determined, and the fusion is performed based on the position information in the same layers in the first sample image and its corresponding color block image, or based on the corresponding pixels of the first sample image and its corresponding color block image, such as superimposing and summing the pixel values ​​at the same position.

[0119] In practical applications, as described above, in order to facilitate data processing, the sample design draft can be parsed into a file format with a hierarchical structure, such as a JSON file, and the design end can be called to convert all the artboards in the sample design draft into corresponding interface images. Afterwards, a layer tree can be constructed based on the hierarchical structure, and the layer tree can be traversed from bottom to top to obtain the spatial information of all layers, etc. All layers can be filled with colors according to the traversal order, wherein different layers corresponding to each interface image have different fill colors, so that a color block map corresponding to each interface image can be generated. Since different layers in the color block map have different colors, the boundary information of different layers in the interface image can be clearly expressed through the color block map. During the traversal process, when the traversed layer corresponds to a layer group with a merge identifier, the layer group is the corresponding sample area, and its spatial information is used as the spatial information of the sample area and as a model label for model training, etc.

[0120] In some embodiments, the area detection network includes a first identification module and at least one second identification module;

[0121] The first sample image feature is used as input data of the region detection network, and the spatial information of at least one sample region is used as a training label. Training the region detection network may include: using the first sample image feature as input data of a first recognition module, and the spatial information of at least one sample region is used as a training label to train the first recognition module;

[0122] The following training operation is performed for any second recognition module: the intersection parameter values ​​of the multiple first bounding boxes determined by the previous recognition module and the at least one sample area are calculated, and at least one second bounding box whose intersection parameter value is greater than the intersection threshold is screened; the image features corresponding to the at least one second bounding box are used as input data of the second recognition module, and the spatial information of the at least one sample area is used as a training label to train the second recognition module; wherein the previous recognition module corresponding to the first second recognition module is the first recognition module; the screening threshold corresponding to the at least one second recognition module is increased in sequence according to the processing order.

[0123] Among them, by calculating the intersection parameter values ​​of the multiple first bounding boxes determined by the previous recognition module and at least one sample area, at least one second bounding box corresponding to each sample area and having a selected intersection parameter value greater than an intersection threshold can be screened and obtained. Specifically, the second recognition model can be trained using the at least one second bounding box corresponding to each sample area and the spatial information of the sample area.

[0124] It can be understood that, among the multiple first bounding boxes corresponding to each sample area, the larger the intersection area with the sample area, the more accurate the first bounding box is. Therefore, at least one second bounding box can be determined according to the intersection parameter value of the sample area and the multiple first bounding boxes, wherein the intersection parameter value can be the intersection area, and further, it can also be an IOU (Intersection over Union) value, that is, the ratio of the intersection and union of each first bounding box and each sample area, wherein the intersection can refer to the intersection area, and the union can refer to the combined area, etc. In the case where the first bounding box completely overlaps with the sample area, the IOU value can be 1.

[0125] Among them, the image features corresponding to at least one second bounding box are used as input data of the second recognition module, and the spatial information of at least one sample area is used as a training label. Training the second recognition module can be: using the image features corresponding to at least one second bounding box as input data of the second recognition module; calculating a third loss value using a second loss function based on the output result of the second recognition module and the spatial information of at least one sample area; and adjusting the model parameters of the second recognition module based on the third loss value.

[0126] The second loss function of each second recognition module is expressed by the following formula:

[0127] L(x t ,g)=L loc (f t (x t ,b t ),g);

[0128] b t =f t-1 (x t-1 ,b t-1 )

[0129] Among them, f t represents the tth second recognition module, x t is the image feature corresponding to at least one second bounding box input to the t-th second recognition module, and g is the training label, that is, the spatial information of the sample area.

[0130] Generally, the detection model includes a plurality of second recognition modules, and the screening threshold corresponding to the input data of each second recognition module is greater than the screening threshold corresponding to the previous recognition module.

[0131] The second recognition model may be implemented using a regressor to perform multi-stage regression processing on the multiple first bounding boxes determined by the first recognition module.

[0132] In some embodiments, the first identification module includes a first proposal module and at least one second proposal module;

[0133] The first sample image feature is used as input data of the first recognition module, and the spatial information of at least one sample area is used as a training label. Training the first recognition module includes: using the first sample image feature as input data of the first proposal module, and performing the following processing operations on any second proposal module: determining at least one third bounding box based on the output result of the previous proposal module; and correcting at least one third bounding box based on the offset parameter to obtain at least one fourth bounding box; and using the image features corresponding to the at least one fourth bounding box as input data of the second proposal module; wherein the previous proposal module of the first second proposal module is the first proposal module; using the spatial information of at least one sample area as a training label, and training the first proposal module and the second proposal module.

[0134] Among them, the first proposal module and the second proposal module can be implemented using an adaptive convolutional network. An offset parameter is added to the second proposal module. The offset parameter can be used as a model parameter and continuously learned and updated during the model training process, so that the first recognition model can accurately extract the target area.

[0135] The step of using the spatial information of at least one sample area as a training label and training the first proposal module and the second proposal module may include:

[0136] Calculating a first regression loss value using a first loss function based on an output result of the first proposal module and spatial information of at least one sample region;

[0137] Calculate a second regression loss value and a first classification loss value using a first loss function based on an output result of at least one second proposal module and spatial information of at least one sample region;

[0138] Based on the first regression loss value, the second regression loss value and the first classification loss value, the model parameters corresponding to the first recognition module and the second recognition module are adjusted respectively.

[0139] Among them, the first loss function can be composed of a regression loss function and a classification loss function, which can be expressed as follows:

[0140]

[0141] in, represents the regression loss function of the τth proposal module, α τ represents the weight coefficient of τ proposed modules, which can be set according to actual conditions. cls is the classification loss function of the first recognition module. λ represents an adjustment coefficient, for example, it can be greater than or equal to 10. In practical applications, the regression loss can use the IOU loss function, and the classification loss can use the cross entropy loss function.

[0142] It should be noted that the above is only an example of a possible implementation architecture of the detection model. In actual applications, the detection model can be implemented by any machine learning model, such as a neural network model, a deep learning model, etc. This application does not specifically limit this.

[0143] The detection model obtained by the above training can be used for layer processing, such as Figure 4 FIG. 1 is a flowchart of another embodiment of a layer processing method provided by an embodiment of the present application. The method may include the following steps:

[0144] 401: Get the target image corresponding to the interface design draft.

[0145] As an optional method, at least one artboard in the interface design draft may be used as an interface picture.

[0146] As an optional method, at least one drawing board in the interface design draft may be converted to generate a corresponding interface image.

[0147] Each interface image can be used as a target image to be processed according to the technical solution of the embodiment of the present application.

[0148] As another optional method, at least one artboard in the interface design draft may be converted to generate a corresponding interface image; then, any interface image may be segmented according to the segmentation requirements to obtain at least one segmented image; and any segmented image may be used as the target image.

[0149] That is, each segmented image can be used as a target image to be processed according to the technical solution of the embodiment of the present application.

[0150] 402: Determine at least one target region in the target image that meets the merging requirement using the detection model.

[0151] The detection model is trained using the spatial information of the first sample image and at least one sample area annotated in the first sample image that meets the merging requirements. The specific training method can be found in the previous text and will not be repeated here.

[0152] 403: Based on the intersection information between the at least one target area and different layers, identify multiple target layers corresponding to the at least one target area.

[0153] The multiple target layers are used for merging processing to serve as a component corresponding to the target area, and the merging processing can be performed during the code generation process.

[0154] The identification process can be performed based on the intersection information of at least one target area and different layers corresponding to the interface design draft or the intersection information of different layers corresponding to the target image. That is, the identification process can be performed from the interface design draft dimension or the target image dimension, and this application does not specifically limit this.

[0155] The intersection information may include intersection distance and / or layer spacing, etc.

[0156] As an optional method, for any target area, the intersection area between the target area and different layers can be calculated, and multiple candidate layers whose intersection areas are greater than the area threshold can be determined, and multiple target layers whose layer spacing meets the distance requirement can be determined from the multiple candidate layers.

[0157] Optionally, multiple layers corresponding to the target image may constitute a first layer list;

[0158] Afterwards, the following screening operations can be performed by selecting one target area in order from the smallest to the largest area of ​​at least one target area:

[0159] Calculate the intersection areas of the target area with different layers in the first layer list respectively, and store the target areas corresponding to multiple candidate layers whose intersection areas are greater than the area threshold in the second layer list; and calculate the layer distance between two adjacent candidate layers in a bottom-up order based on the hierarchical structure of the multiple candidate layers; and if the layer distance between two adjacent candidate layers is greater than the distance threshold, filter out the next layer of the two adjacent candidate layers from the second layer list; and update the first layer list according to the second layer list; if the screening operations for at least one target area are all completed, the candidate layer in the second layer list is the target layer.

[0160] It is understandable that each target area may correspond to a component, which is obtained by editing multiple target layers. The target area may refer to the smallest area in the target image that contains the multiple target layers. Therefore, multiple candidate layers corresponding to the target area can be determined based on the intersection area between the target area and different layers. Since the target area corresponding to some components is relatively large, the multiple candidate layers corresponding to the target area determined based on the intersection area between the target area and different layers may contain layers that do not belong to the target area. For example, the area of ​​the target area corresponding to the background component is relatively large, and the multiple candidate layers corresponding to it may include text layers, and the text layers do not belong to the background component. Therefore, the text layers need to be filtered out. Since the layer distances between multiple candidate layers belonging to the same component are all within the preset threshold range, the multiple target layers corresponding to the target area can be determined based on the layer spacing of the multiple candidate layers corresponding to the target area.

[0161] As another optional method, based on the intersection information between at least one target area and different layers, identifying at least one target layer corresponding to at least one target area can be: based on the intersection information between at least one target area and different layers, using a recognition model to identify at least one target layer corresponding to at least one target area.

[0162] The recognition model is obtained by training using the sample area of ​​the sample image and multiple target layers corresponding to the sample area. That is, the model can be used for end-to-end recognition, and at least one target layer corresponding to at least one target area can be identified based on the at least one target area. The recognition model can be a deep learning model, etc.

[0163] In this embodiment, for the target image corresponding to the interface design draft; use the detection model to determine at least one target area in the target image that meets the merging requirements; the detection model is trained using the spatial information of the first sample image and at least one sample area annotated by the first sample image that meets the merging requirements; based on the intersection information of at least one target area with different layers, identify multiple target layers corresponding to at least one target area; wherein the multiple target layers are used for merging processing. The technical solution provided in the embodiment of the present application uses the detection model to determine at least one target area in the target image that meets the merging requirements, thereby determining the target layer corresponding to the target area based on the intersection information of the target area with different layers, thereby improving the efficiency and accuracy of layer search. The multiple target layers corresponding to the target area can be merged and processed during the intelligent code development process, and converted into the same component for processing, thereby improving the quality of code generation.

[0164] In some embodiments, the method may further include:

[0165] Determine a color block image generated by color filling of multiple layers corresponding to the target image; wherein different layers are filled with different colors;

[0166] The step of determining at least one target region of the target image by using the detection model may include:

[0167] The detection model is used to extract image features from the target image and the color block image of the target image, and at least one target area of ​​the target image is identified based on the image features.

[0168] The detection model can be specifically trained using sample image features extracted from the first sample image and the color block image corresponding to the first sample image, as well as spatial information of at least one sample area annotated by the first sample image corresponding to multiple target layers.

[0169] There are multiple possible implementation methods for extracting image features from the target image and the color block image of the target image using the detection model: As a possible implementation method, the color block image and the target image may be first fused to obtain a fused image, and then the detection model may be used to extract image features from the fused image;

[0170] As another possible implementation manner, the detection model may be used to extract a first feature from the target image and a second feature from a color block image of the target image, and the first feature and the second feature may be fused to obtain the image feature.

[0171] The first feature and the second feature may be fused by cumulative summation or weighted summation to obtain the image feature.

[0172] In some embodiments, the interface design draft can be parsed into a file format with a hierarchical structure, and the design end can be called to convert all the artboards in the interface design draft into corresponding interface images. Afterwards, a layer tree can be constructed based on the hierarchical structure, and the layer tree can be traversed from bottom to top. All layers can be filled with colors according to the traversal order, wherein different layers corresponding to each interface image are filled with different colors, so that a color block map corresponding to each interface image can be generated. Therefore, the color block map can be obtained as follows:

[0173] Parse the interface design draft and build a layer tree according to the hierarchical structure of each interface image;

[0174] Traverse the layer tree from bottom to top, fill each layer obtained through the traversal with color, and generate a color block map corresponding to each interface image.

[0175] When the target image is an interface image, the color block image corresponding to the interface image is the color block image of the target image.

[0176] When the target image is a segmented image obtained by segmenting the interface image, the color block image of the interface image can be segmented according to the segmentation method of the interface image, and the corresponding color block image can be determined according to the corresponding segmented part of the target image.

[0177] The layer tree may be a file in a corresponding JSON format.

[0178] As can be seen from the foregoing description, the detection model may include a feature extraction network and a region detection network. Specifically, the feature extraction network in the detection model may be used to extract image features from a target image and a color block map of the target image; and the region detection network in the detection model may be used to identify at least one target region in the target image that meets the merging requirements based on the image features.

[0179] The region detection network may include a first recognition module and at least one second recognition module. Therefore, in some embodiments, identifying at least one target region in the target image that meets the merging requirements based on the image features may include: using the first recognition module in the detection model to identify at least one candidate region of the target image based on the image features; using at least one second recognition module to identify and obtain at least one target region in the target image that meets the merging requirements based on the image features corresponding to the at least one candidate region.

[0180] Optionally, using at least one second recognition module based on the picture features corresponding to at least one candidate area respectively to obtain at least one target area comprises: using each second recognition module based on the picture features corresponding to at least one candidate area recognized by a previous recognition module to recognize at least one candidate area in the target image; wherein the previous recognition module of the first second recognition module is the first recognition module;

[0181] At least one candidate region determined by the last second recognition module is used as a target region in the target picture.

[0182] The first recognition module may include a first proposal module and at least one second proposal module. In some embodiments, using the first recognition module in the detection model, identifying at least one candidate region of the target image based on the image features may include:

[0183] Using a first proposal module to identify and obtain at least one candidate region based on the image features;

[0184] The following processing operations are performed for any second proposal module:

[0185] At least one candidate region determined by a previous proposal module is corrected based on the offset parameter of the second proposal module to obtain at least one corrected region, and the second proposal module is used to identify at least one candidate region based on the picture features corresponding to the at least one corrected region.

[0186] The previous proposal module of the first second proposal module is the first proposal module.

[0187] The at least one candidate region determined by the last second proposal module is the at least one candidate region determined by the first recognition module.

[0188] In some embodiments, after identifying and obtaining multiple target layers corresponding to at least one target area, the method may further include: treating the multiple target layers corresponding to each target area in the interface design draft as a group of layer groups and setting a merge flag to update the interface design draft.

[0189] Different layer groups correspond to different merge identifiers; the layer group is used to encode as a component in the code generation operation.

[0190] In addition, the method may further include:

[0191] Generate target code based on the interface design draft or target image encoding, and during the encoding process, encode the layer group with the merge identifier as a component.

[0192] Alternatively, the interface design draft is sent to the code generation end, so that the code generation end generates the target code based on the interface design draft or the target image encoding, and during the encoding process, the layer group with the merge identifier is encoded as a component.

[0193] In some embodiments, after identifying and obtaining a plurality of target layers corresponding to at least one target area, the method may further include:

[0194] Generate target code based on the interface design draft or target image encoding, and during the encoding process, encode multiple target layers corresponding to each target area as a component.

[0195] That is, when the target code is generated by the processing end, after identifying and obtaining multiple target layers corresponding to at least one target area, the multiple target layers corresponding to each target area can be directly encoded as a component during the encoding process.

[0196] In some embodiments, after identifying and obtaining multiple target layers corresponding to at least one target area, the method may further include: merging multiple icon layers corresponding to each target area in the interface design draft into one layer to update the interface design draft. This makes it possible to improve the quality of the generated code by excluding the target layers that need to be merged in the interface design draft.

[0197] Among them, the above-mentioned target code generated based on the interface design draft or the target image coding can be implemented by using an intelligent code generation algorithm, for example, a code generation model can be used to intelligently generate different types of available codes based on the interface design draft or the target image. The code generation module can be implemented by using deep learning technology, for example, and can be pre-trained based on a sample design draft or a sample image, and the corresponding target code, etc. This application does not specifically describe the intelligent code generation algorithm.

[0198] In order to further understand the technical solution of the embodiment of the present application, Figure 5 The scene interaction diagram shown and Figure 6 The model processing schematic diagram shown describes the technical solution of the embodiment of the present application in detail.

[0199] It should be noted that this embodiment uses a drawing board in an interface design draft as an example to illustrate the technical solution. The interface design draft may be in Sketch format, for example.

[0200] like Figure 5 As described above, the interface design draft generated by the design end 501 can be sent to the processing end 502 .

[0201] The processing end 502 can parse the interface design draft, convert the artboards in the interface design draft into interface images, and use each interface image as a target image, or use segmented images obtained by segmenting each interface image as a target image.

[0202] To facilitate data processing, when the interface design draft is in Sketch format, for example, the interface design draft can be converted into a different format. For example, a JSON file with a hierarchical structure corresponding to the interface design draft can be obtained, and the interface provided by Sketch can be called to convert the artboard in the design draft into a corresponding interface image.

[0203] According to the JSON file, the multiple layers involved in each interface image can be converted into a layer tree according to the hierarchical structure, and the layer tree can be traversed from bottom to top to obtain the spatial information of each layer, and each layer obtained by traversal can be filled with color and reorganized to obtain the color block map corresponding to each interface image. According to the above-mentioned method of determining the target image, the color block map determined by the target image can be determined.

[0204] Afterwards, for the target image, the detection model can be used to determine at least one target area in the target image that meets the merging requirements based on the target image and the color block image. Specifically, Figure 6 In the model processing schematic diagram shown, as an optional method, the target image 60 and the corresponding color block image 61 can be fused to obtain a fused image 62 with boundary features, and the fused image is input into the feature extraction network 601 of the detection model to obtain image features. The image features can be input into the region detection network 602, and based on the output result of the region detection network 602, at least one target region 63 ( Figure 6 A target area is used as an example for explanation. It should be noted that, in actual applications, one or more target areas may be extracted from a target image. The at least one target area 63 is an area of ​​the target image that needs to be merged, such as the area where the same component expressed by multiple fragmented layers is located.

[0205] Among them, Figure 6 As shown, the region detection network can be composed of a first recognition module and at least one second recognition module, and the first recognition module can be composed of a first proposal model and at least one second proposal model. The detailed structural description can be found in the above embodiments and will not be repeated here. The image features obtained by the feature extraction network can be input into the first proposal module in the first recognition module, and the first proposal module is used to identify and obtain at least one candidate region based on the image features; for any second proposal module, at least one candidate region determined by the previous proposal module is corrected based on the offset parameters of the second proposal module to obtain at least one corrected region, and the second proposal module is used to identify and obtain at least one candidate region based on the image features corresponding to the at least one corrected region; wherein, the previous proposal module of the first second proposal module is the first proposal module.

[0206] The image features corresponding to at least one candidate area determined by the first recognition module can be input into the first second recognition module, so that each second recognition module can identify at least one candidate area in the target image based on the image features corresponding to at least one candidate area identified by the previous recognition module; wherein the previous recognition module of the first second recognition module is the first recognition module; and the at least one candidate area determined by the last second recognition module is respectively used as the target area in the target image.

[0207] After determining at least one target area 63, based on the intersection information between the at least one target area 63 and different layers in the target image, multiple target layers 64 corresponding to each target area 63 can be determined, wherein the method for determining the target layer is described in detail in the corresponding embodiment above and will not be repeated here.

[0208] After the processing end 502 determines the multiple target layers corresponding to each target area, the multiple target layers corresponding to each target area in the interface design draft can be taken as a layer group and a merge flag can be set to update the interface design draft.

[0209] The processing end 502 may send the updated interface design draft to the code generation end 503 .

[0210] The code generation terminal 503 can generate target code based on the interface design draft code, and during the coding process, encode the layer group with the merge identifier as a component.

[0211] Figure 7 A schematic diagram of the structure of an embodiment of the layer processing device provided by the present application is shown, which is applied to the server, such as Figure 7 The device comprises: a determination module 71 and an identification module 72 .

[0212] The determination module 71 is used to obtain a target image corresponding to the interface design draft; and determine at least one target area in the target image that meets the merging requirements;

[0213] The identification module 72 is used to identify multiple target layers corresponding to the at least one target area based on the intersection information between the at least one target area and different layers, wherein the multiple target layers are used for merging processing.

[0214] In some embodiments, the determination module 71 is further used to determine a color block image generated by color filling of multiple layers corresponding to the target image; wherein different layers are filled with different colors.

[0215] In some embodiments, the determination module 71 is specifically used to extract image features from the target image and the color block image of the target image using a detection model, and identify at least one target area of ​​the target image based on the image features.

[0216] In some embodiments, the determination module 71 is further specifically used to fuse the color block map with the target image to obtain a fused image, and to extract image features from the fused image using the detection model; or, to extract a first feature from the target image and a second feature from the color block map of the target image using the detection model, and to fuse the first feature and the second feature to obtain image features.

[0217] In some embodiments, the determination module 71 is further specifically used to use the first recognition module in the detection model to identify at least one candidate area of ​​the target image based on the image features; and use at least one second recognition module to identify at least one target area in the target image based on the image features corresponding to the at least one candidate area.

[0218] In some embodiments, the determination module 71 is further specifically used to identify at least one candidate area in the target image by using each second recognition module based on the image features corresponding to at least one candidate area recognized by the previous recognition module; wherein the previous recognition module of the first second recognition module is the first recognition module; and the at least one candidate area determined by the last second recognition module is respectively used as the target area in the target image.

[0219] In some embodiments, the determination module 71 is further specifically used to use the first proposal module to identify and obtain at least one candidate area based on the image features; perform the following processing operations for any second proposal module: correct at least one candidate area determined by the previous proposal module based on the offset parameters of the second proposal module to obtain at least one corrected area, and use the second proposal module to identify and obtain at least one candidate area based on the image features corresponding to the at least one corrected area; wherein the previous proposal module of the first second proposal module is the first proposal module.

[0220] In some embodiments, the identification module 72 is specifically used to calculate, for any target area, the intersection areas of the target area with different layers, and determine multiple candidate layers whose intersection areas are greater than an area threshold, and determine multiple target layers whose layer spacing meets the distance requirement from the multiple candidate layers.

[0221] In some embodiments, the identification module 72 is further specifically used to determine a first layer list composed of multiple layers corresponding to the target image; select a target area in turn in the order of the area area of ​​the at least one target area from small to large to perform the following screening operations: calculate the intersection area of ​​the target area with different layers in the first layer list respectively, and store multiple candidate layers whose intersection area is greater than the area threshold into a second layer list corresponding to the target area; based on the hierarchical structure of the multiple candidate layers, calculate the layer distance between two adjacent candidate layers in a bottom-up order; if the layer distance between two adjacent candidate layers is greater than the distance threshold, filter out the next layer of the two adjacent candidate layers from the second layer list; update the first layer list according to the second layer list; after all the screening operations for the at least one target area are completed, determine that the candidate layer in the second layer list is the target layer.

[0222] In some embodiments, the recognition module 72 is also specifically used to identify at least one target layer corresponding to the at least one target area based on the intersection information between the at least one target area and different layers, using a recognition model; wherein the recognition model is trained using the sample area of ​​the sample image and multiple target layers corresponding to the sample area.

[0223] In some embodiments, the determination module 71 is also specifically used to convert at least one drawing board in the interface design draft to generate a corresponding interface image; segment any interface image according to the segmentation requirements to obtain at least one segmented image; and use any segmented image as the target image.

[0224] In some embodiments, the above apparatus further comprises:

[0225] The deleting module is used to delete any one of the two segmented pictures that meet the similarity requirement according to the picture similarity to update the at least one segmented picture.

[0226] In some embodiments, the determination module 71 is further specifically used to convert at least one drawing board in the interface design draft to generate a corresponding interface image; and use any interface image as a target image.

[0227] In some embodiments, the determination module 71 is further specifically used to parse the interface design draft, build a layer tree according to the hierarchical structure of each interface image; traverse the layer tree in a bottom-up order, and fill each layer obtained by the traversal with color to generate a color block map corresponding to each interface image.

[0228] In some embodiments, the above apparatus further comprises:

[0229] The filtering module is used to filter out the interface images that do not meet the image requirements from at least one interface image corresponding to the interface design draft.

[0230] In some embodiments, the identification module 72 is also used to identify at least one target layer corresponding to the at least one target area based on intersection information between the at least one target area and different layers corresponding to the interface design draft; or to identify at least one target layer corresponding to the at least one target area based on intersection information between the at least one target area and different layers corresponding to the target image.

[0231] In some embodiments, the above apparatus further comprises:

[0232] An update module is used to treat multiple target layers corresponding to each target area in the interface design draft as a group of layer groups and set a merge identifier to update the interface design draft; wherein the merge identifiers corresponding to different layer groups are different; and the layer group is used to be encoded as a component in the code generation operation.

[0233] The generation module is used to respond to the code generation instruction, compile and generate the target code based on the interface design draft or the target image, and during the compilation process, encode the layer group with the merge identifier as a component.

[0234] In some embodiments, the update module is also used to treat multiple target layers corresponding to each target area in the interface design draft as a group of layer groups and set a merge identifier to update the interface design draft; wherein, the merge identifiers corresponding to different layer groups are different; and the layer group is used to be encoded as a component in the code generation operation.

[0235] The generation module is also used to respond to the code generation instruction, compile and generate the target code based on the interface design draft or the target image, and during the compilation process, encode the layer group with the merge identifier as a component.

[0236] In some embodiments, the determination module 71 is specifically used to use a detection model to determine at least one target area in the target image that meets the merging requirements; the detection model is trained using the spatial information of the first sample image and at least one sample area annotated in the first sample image that meets the merging requirements.

[0237] Figure 7 The layer processing device can execute Figure 2 or Figure 4The implementation principle and technical effect of the layer processing method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the layer processing device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0238] Figure 8 A schematic diagram of the structure of an embodiment of the model generation device provided by the present application is shown, which is applied to a client, such as Figure 8 The device comprises: an acquisition module 81 and a training module 82.

[0239] An acquisition module 81 is configured to acquire a first sample image and spatial information of at least one sample area marked in the first sample image and meeting the merging requirements; the at least one sample area corresponds to a plurality of target layers respectively;

[0240] A training module 82, configured to train a detection model by using the first sample image as input data and the spatial information of the at least one sample region as a training label;

[0241] Among them, the detection model is used to determine at least one target area in the target image that meets the merging requirements based on the target image corresponding to the interface design draft; the at least one target area is used to identify multiple target layers corresponding to each other based on intersection information with different layers respectively; the multiple target layers are used for merging processing to serve as a component corresponding to the target area.

[0242] In some embodiments, the acquisition module 81 is specifically used to convert at least one artboard in the sample design draft to generate a corresponding interface image; use any interface image as the first sample image; or segment any interface image according to the segmentation requirements to obtain at least one segmented image, and use any segmented image as the first sample image; determine at least one layer group with a merge identifier set in the multiple layers corresponding to the first sample image; the at least one layer group respectively includes multiple target layers; and use the spatial information corresponding to any layer group as the spatial information of a sample area that meets the merge requirements.

[0243] In some embodiments, the above apparatus further comprises:

[0244] A deletion module is used to filter out interface images that do not meet the image requirements from at least one interface image corresponding to the sample design draft; delete a predetermined proportion of layers from multiple layers in any interface image that do not have a merge mark set to obtain an extended image; use the extended image as the first sample image; or segment the extended image according to the segmentation requirements to obtain at least one segmented image, and use any segmented image as the first sample image; delete any segmented image of the two segmented images that meet the similarity requirements based on the image similarity; delete a predetermined proportion of layers that meet the deletion conditions from multiple layers in any interface image that do not have a merge mark set to obtain an extended image.

[0245] In some embodiments, the training module 82 is specifically used for the detection model including a feature extraction network and a region detection network; the first sample image is used as input data and the spatial information of the at least one sample region is used as a training label, and the training detection model includes: inputting the first sample image into the feature extraction network to extract the first sample image feature; using the first sample image feature as the input data of the region detection network, and the spatial information of the at least one sample region is used as a training label to train the region detection network.

[0246] In some embodiments, the training module 82 is further specifically used for the region detection network to include a first recognition module and at least one second recognition module; the first sample image feature is used as the input data of the region detection network, and the spatial information of the at least one sample area is used as a training label, and the training of the region detection network includes: using the first sample image feature as the input data of the first recognition module, and the spatial information of the at least one sample area as a training label, to train the first recognition module; performing the following training operations for any second recognition module: calculating the intersection parameter values ​​of the multiple first bounding boxes determined by the previous recognition module and the at least one target area, respectively, and screening at least one second bounding box whose intersection parameter value is greater than the intersection threshold; using the image features corresponding to the at least one second bounding box as the input data of the second recognition module, and using the spatial information of the at least one sample area as a training label, to train the second recognition module; wherein, the previous recognition module corresponding to the first second recognition module is the first recognition module; the screening threshold corresponding to the at least one second recognition module increases in sequence according to the processing order.

[0247] In some embodiments, the training module 82 is further specifically used for the first recognition module to include a first proposal module and at least one second proposal module; the first sample image feature is used as the input data of the first recognition module, and the spatial information of the sample area is used as the training label, and the training of the first recognition module includes: using the first sample image feature as the input data of the first proposal module, and performing the following processing operations for any second proposal module: determining at least one third bounding box based on the output result of the previous proposal module; and correcting the at least one third bounding box based on the offset parameter to obtain at least one fourth bounding box; and using the image features corresponding to the at least one fourth bounding box as the input data of the second proposal module; wherein the previous proposal module of the first second proposal module is the first proposal module; using the spatial information of the at least one sample area as the training label, and training the first proposal module and the second proposal module.

[0248] In some embodiments, the training module 82 is further specifically used to calculate a first regression loss value using a first loss function based on the output result of the first proposal module and the spatial information of the at least one sample area; calculate a second regression loss value and a first classification loss value using the second loss function based on the output result of the at least one second proposal module and the spatial information of the at least one sample area; and adjust the model parameters corresponding to the first processing module and the second recognition module respectively based on the first regression loss value, the second regression loss value and the first classification loss value.

[0249] In some embodiments, the training module 82 is further specifically used to use the image features corresponding to the at least one second bounding box as input data of the second recognition module; based on the output result of the second recognition module and the spatial information of the at least one sample area, calculate the third loss value using a third loss function; based on the third loss value, adjust the model parameters of the second recognition module.

[0250] In some embodiments, the training module 82 is further specifically used to train the feature extraction network using the second sample image and the corresponding second sample image features.

[0251] Figure 8 The model generation device can execute Figure 4 The implementation principle and technical effect of the model generation method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the model generation device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0252] The present application also provides a computing device, such as Fig. 9 As shown, the device may include a storage component 901 and a processing component 902;

[0253] The storage component stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 902 to implement as a computing device Figure 2 or Figure 4 The layer processing method or Figure 3 The model generation method described.

[0254] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0255] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0256] The processing component may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0257] The storage component 901 is configured to store various types of data to support operations at the terminal. The storage component 901 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0258] It should be noted that the above computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. It can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0259] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 2 or Figure 4 The file processing method or Figure 3The model generation method. The computer readable medium may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device.

[0260] The present application also provides a computer program product, which includes a computer program carried on a computer-readable storage medium. When the computer program is executed by a computer, the above-mentioned Figure 2 or Figure 4 The file processing method or Figure 3 The model generation method described. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, various functions defined in the system of the present application are executed.

[0261] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0262] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0263] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0264] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A layer processing method, characterized in that: include: Get the target image corresponding to the interface design draft; Determine at least one target area in the target image that meets the merging requirements; Based on the intersection information between the at least one target area and different layers, identifying a plurality of target layers whose intersection information respectively corresponds to the at least one target area meets a merging condition, wherein the plurality of target layers are used for merging processing to serve as a component corresponding to the target area; The intersection information includes an intersection area and / or a layer spacing, and the merging condition includes that the intersection area is greater than an area threshold and / or the layer spacing meets a distance requirement.

2. The method according to claim 1, characterized in that Also includes: Determine a color block image generated by color filling of multiple layers corresponding to the target image; wherein different layers are filled with different colors; The determining at least one target area in the target image that meets the merging requirement comprises: Extracting image features from the target image and the color block map of the target image, and identifying at least one target area in the target image that meets the merging requirements based on the image features.

3. The method according to claim 2, characterized in that The extracting picture features from the target picture and the color block map of the target picture includes: Fusing the color block image with the target image to obtain a fused image, and extracting image features from the fused image; Alternatively, a first feature is extracted from the target image and a second feature is extracted from a color block image of the target image, and the first feature and the second feature are fused to obtain an image feature.

4. The method according to claim 1, characterized in that: The step of identifying, based on the intersection information between the at least one target area and different layers, a plurality of target layers whose intersection information respectively corresponds to the at least one target area and satisfies a merging condition comprises: For any target area, the intersection areas of the target area with different layers are calculated, and multiple candidate layers whose intersection areas are greater than an area threshold are determined, and multiple target layers whose layer spacing meets the distance requirement are determined from the multiple candidate layers.

5. The method according to claim 4, characterized in that The method of determining the multiple target layers includes: Determine a first layer list consisting of multiple layers corresponding to the target image; According to the order of the area area of ​​the at least one target area from small to large, one target area is selected in turn to perform the following screening operation: Calculating the intersection areas of the target area and different layers in the first layer list respectively, and storing multiple candidate layers whose intersection areas are greater than an area threshold in a second layer list corresponding to the target area; Based on the hierarchical structure of the plurality of candidate layers, calculating the layer spacing between two adjacent candidate layers in order from bottom to top; If the layer distance between two adjacent candidate layers is greater than the distance threshold, filtering out the next layer between the two adjacent candidate layers from the second layer list; Update the first layer list according to the second layer list; After all the screening operations for the at least one target area are completed, the candidate layer in the second layer list is determined to be the target layer.

6. The method according to claim 1, characterized in that The step of identifying, based on the intersection information between the at least one target area and different layers, a plurality of target layers whose intersection information respectively corresponds to the at least one target area and satisfies a merging condition comprises: Based on the intersection information between the at least one target area and different layers, using a recognition model to identify at least one target layer corresponding to the at least one target area; The recognition model is obtained by training using any sample area of ​​the sample image and multiple target layers corresponding to the sample area.

7. The method according to claim 1, characterized in that The method further comprises: In the interface design draft, multiple target layers corresponding to each target area are taken as a group of layer groups and a merge flag is set to update the interface design draft; the layer group is used to be encoded as a component in the code generation operation; Alternatively, the multiple target layers corresponding to each target area in the interface design draft are taken as a group of layer groups and a merge identifier is set to update the interface design draft; and in response to a code generation instruction, the target code is compiled based on the interface design draft or the target image, and during the compilation process, the layer group set with a merge identifier is encoded as a component.

8. The method according to claim 1, characterized in that: The determining at least one target area in the target image that meets the merging requirement comprises: A detection model is used to determine at least one target area in the target image that meets the merging requirements; the detection model is trained using a sample image and spatial information of at least one sample area annotated in the sample image that meets the merging requirements.

9. A model generation method, characterized in that: include: Acquire a first sample image and spatial information of at least one sample area marked in the first sample image and meeting the merging requirement; The at least one sample area corresponds to a plurality of target layers respectively; the spatial information includes position information and size information; Using the first sample image as input data and the spatial information of the at least one sample area as a training label, training a detection model; Among them, the detection model is used to determine at least one target area in the target image that meets the merging requirements based on the target image corresponding to the interface design draft; the at least one target area is used to identify multiple target layers whose corresponding intersection information meets the merging conditions based on the intersection information with different layers respectively; the multiple target layers are used for merging processing to serve as a component corresponding to the target area; the intersection information includes the intersection area and / or the layer spacing, and the merging conditions include the intersection area being greater than the area threshold and / or the layer spacing meeting the distance requirements.

10. The method according to claim 9, characterized in that The obtaining of the first sample image and the spatial information of at least one sample area marked by the first sample image and meeting the merging requirement comprises: Convert at least one artboard in the sample design draft to generate a corresponding interface image; Using any interface image as the first sample image; or segmenting any interface image according to the segmentation requirement to obtain at least one segmented image, and using any segmented image as the first sample image; Determine at least one layer group having a merge flag set among the multiple layers corresponding to the first sample image; the at least one layer group includes multiple target layers respectively; The spatial information corresponding to any layer group is taken as the spatial information of a sample area that meets the merging requirements.

11. The method according to claim 10, characterized in that The method further comprises: Deleting a predetermined proportion of layers from a plurality of layers for which no merge mark is set in any interface image, so as to obtain an extended image; The extended picture is used as the first sample picture; or the extended picture is segmented according to the segmentation requirement to obtain at least one segmented picture, and any one of the segmented pictures is used as the first sample picture.

12. The method according to claim 9, characterized in that The detection model includes a feature extraction network and a region detection network; The training of the detection model by using the first sample image as input data and the spatial information of the at least one sample area as a training label comprises: Inputting the first sample image into a feature extraction network to extract features of the first sample image; The first sample image feature is used as input data of the region detection network, and the spatial information of the at least one sample region is used as a training label to train the region detection network.

13. The method according to claim 12, characterized in that The area detection network includes a first identification module and at least one second identification module; The using the first sample image feature as input data of the region detection network and the spatial information of the at least one sample region as a training label, and training the region detection network comprises: Using the first sample image feature as input data of the first recognition module and the spatial information of the at least one sample area as a training label to train the first recognition module; The following training operations are performed for any second recognition module: Calculating intersection parameter values ​​of the plurality of first bounding boxes determined by the previous recognition module and the at least one target area, and selecting at least one second bounding box whose intersection parameter value is greater than an intersection threshold; Using the picture features respectively corresponding to the at least one second bounding box as input data of the second recognition module, and using the spatial information of the at least one sample area as a training label, to train the second recognition module; The previous recognition module corresponding to the first second recognition module is the first recognition module; and the screening thresholds corresponding to the at least one second recognition module are increased in sequence according to the processing order.

14. The method according to claim 13, characterized in that The first identification module includes a first proposal module and at least one second proposal module; The first sample image feature is used as input data of the first recognition module, and the spatial information of the sample area is used as a training label, and training the first recognition module includes: The first sample image feature is used as input data of the first proposal module, and the following processing operations are performed on any second proposal module: Based on the output result of the previous proposal module, at least one third bounding box is determined; and based on the offset parameter, the at least one third bounding box is corrected to obtain at least one fourth bounding box; and the picture features corresponding to the at least one fourth bounding box are used as input data of the second proposal module; wherein the previous proposal module of the first second proposal module is the first proposal module; The spatial information of the at least one sample area is used as a training label to train the first proposal module and the second proposal module.

15. A computing device, characterized in that: It comprises a storage component and a processing component; the storage component stores one or more computer program instructions; the one or more computer program instructions are called and executed by the processing component to implement the layer processing method as described in any one of claims 1 to 8 or the model generation method as described in any one of claims 9 to 14.

Images