Data processing method, device, computer equipment and computer readable storage medium

Abstract

The present disclosure provides a data processing method, device, computer equipment and computer-readable storage medium, which belongs to the field of data visualization. The method includes: sending a first image to a server through a computer device, the server identifying the first image to obtain the boundary information of the module in the first image, the boundary information including the position of the module, the server identifying the visual component in the module to obtain the category of the visual component, and sending the boundary information of the module and the category of the visual component to the computer device. The above process can directly identify the image containing the design draft, thereby obtaining the module division in the image and the category of the visual component in the module, and automatically generating the screen layout based on the recognition result, without the need for manual operation by the user, thereby reducing the waste of time cost.

Description

Technical Field

[0001] The present disclosure relates to the field of data visualization, and in particular to a data processing method, apparatus, computer device, and computer-readable storage medium. Background Art

[0002] With the booming development of the big data industry, more and more companies are beginning to realize the importance of data management and application, especially the use of data visualization technology to display data. Data visualization is the visualization of data in the database. It is the application of visualization technology in the field of non-spatial data. It uses graphics or images to display data and its structural relationships in a more intuitive way. Its basic idea is to treat a single data item in the database as a single primitive, and to form a data image from a large number of data sets. At the same time, the various attribute values ​​of the data are represented in multiple dimensions to observe and analyze the data more deeply. Through data visualization, the process of understanding multi-dimensional data is transformed into a simple process of looking at colors, distinguishing lengths, and distinguishing heights, which greatly shortens the time required to understand data. Therefore, more and more companies are beginning to use data visualization technology to display, monitor and analyze various types of operational data.

[0003] At present, in the process of using data visualization technology, the user first draws a design on paper to obtain a hand-drawn design, which may include multiple visual components, and then creates a canvas in the data visualization system. According to the visual components designed in the hand-drawn design, the corresponding visual components are moved to the canvas by manually dragging from the visual component library preset in the data visualization system. When it is determined that the generated component layout meets the design requirements, the data is displayed according to the component layout.

[0004] In the above data processing process, the entire component layout is completed by manual dragging and dropping, which has cumbersome operation steps and easily leads to a waste of time costs. Summary of the invention

[0005] The embodiments of the present disclosure provide a data processing method, apparatus, computer device and computer-readable storage medium. The technical problem that the operation steps in the related art are complicated and easily lead to waste of time cost is solved. The technical solution is as follows:

[0006] In a first aspect, a data processing method is provided, the method comprising:

[0007] Acquire a first image, where the first image is an image of a screen design diagram;

[0008] Recognize the first image to obtain boundary information of a module, wherein the module includes at least one visual component, and the boundary information includes a module position;

[0009] Identifying the visual components in the module to obtain the categories of the visual components;

[0010] At a position corresponding to the boundary information of the module in the canvas, the visual component is created according to the category of the visual component in the module to obtain a screen layout of the target screen.

[0011] In the above implementation process, a computer device sends a first image to a server, and the server identifies the first image to obtain boundary information of the module in the first image, and the boundary information includes the position of the module. The server identifies the visual component in the module, obtains the category of the visual component, and sends the boundary information of the module and the category of the visual component to the computer device. The above process can directly identify the image containing the design draft, thereby obtaining the module division in the image and the category of the visual component in the module, and automatically generates the screen layout based on the identification result, without the need for manual operation by the user, thereby reducing the waste of time cost.

[0012] In a first possible implementation manner of the first aspect, recognizing the first image to obtain boundary information of the module includes:

[0013] Binarizing the first image to obtain a binary image;

[0014] Performing a dilation operation on black pixels in the binary image;

[0015] A black outline in the binary image after the dilation operation is extracted, and a position of the black outline that meets a preset outline condition in the first image is used as boundary information of the module.

[0016] The disclosed embodiment provides a method for identifying a module in a first image to obtain the boundary information of the module, which is beneficial for automatically generating visual components in the module according to the position of the module's boundary information on the canvas, thereby obtaining the screen layout of the target screen.

[0017] In a second possible implementation manner of the first aspect, the black outline that meets the preset outline conditions means: the area of ​​the black outline meets the preset area conditions, the black outline is the outermost outline among multiple nested outlines, and the shape of the black outline is a quadrilateral.

[0018] By limiting the black contour, it is ensured that the extracted black contour is a black contour that meets the preset contour conditions, so as to reduce the waste of time cost in the black contour extraction process.

[0019] In a third possible implementation manner of the first aspect, before using the position of the black outline that meets the preset outline condition in the first image as the boundary information of the module, the method further includes:

[0020] Normalizing the module to obtain boundary information of the normalized module;

[0021] The sending the boundary information of the module and the category of the visual component to the computer device comprises:

[0022] The normalized module boundary information and the category of the visual component are sent to the computer device.

[0023] By normalizing the modules in the first image, more accurate module boundary information can be obtained, thereby saving time for processing the modules and speeding up the generation of the picture layout.

[0024] In a fourth possible implementation manner of the first aspect, the normalizing the module includes at least one of the following:

[0025] Adjusting the boundary of the module to a regular quadrilateral;

[0026] Stretching the boundary of the module so that the module fills the canvas;

[0027] When there is an overlapping area between any two modules, the overlapping area between the any two modules is eliminated.

[0028] By normalizing the modules so that they are regular quadrilaterals that fill the entire canvas and do not overlap each other, the visual components in subsequent modules can be displayed better on the canvas, thereby obtaining a better target screen layout.

[0029] In a fifth possible implementation manner of the first aspect, identifying the visual component in the module to obtain the category of the visual component includes:

[0030] The image block corresponding to the module in the first image is input into a visual component classification model, and the image block is identified by the visual component classification model to obtain the category of the visual component in the module.

[0031] The visual component classification model recognizes the image blocks corresponding to the modules in the first image and obtains the categories of the visual components in the modules, thereby ensuring that the visual components displayed in the subsequent screen layout are correct and reducing the waste of time costs in the screen layout generation process.

[0032] In a sixth possible implementation manner of the first aspect, after the image block is identified by the visual component classification model to obtain the category of the visual component in the module, the method further includes:

[0033] When a feedback result on the category of the visual component in the module is received, the feedback result is sent to the server.

[0034] The server updates the visual component classification model based on the feedback result of the category of the visual component to improve the accuracy of the visual component classification model in classifying the visual component.

[0035] In a seventh possible implementation manner of the first aspect, the identifying the first image to obtain the boundary information of the module includes:

[0036] Based on the valid area of ​​the first image, obtain a second image, where the second image includes image content within the valid area;

[0037] The second image is recognized to obtain boundary information of the module, where the boundary information includes the module position.

[0038] When the first image is an image that includes areas outside the effective area, the effective area of ​​the first image needs to be identified to obtain the second image, so as to improve the recognition accuracy of the modules in the image.

[0039] In an eighth possible implementation manner of the first aspect, before obtaining the second image based on the valid area of ​​the first image, the method further includes any one of the following:

[0040] Based on the pixel value range corresponding to the white pixel points, determine the target contour in the first image, the target contour is the largest contour among the contours composed of the white pixel points in the first image, and take the area within the target contour as the valid area;

[0041] Perform edge detection on the first image to obtain at least one region, and take a region in the at least one region whose region shape meets the target condition as the valid region;

[0042] The first image is binarized to obtain at least one white area, and a white area in the at least one white area whose area shape meets the target condition is used as the effective area.

[0043] Based on the above three effective area recognition methods, the effective area recognition process of the image including the effective area can be realized, and then the modules in the effective area can be recognized to obtain more accurate module boundary information.

[0044] In a ninth possible implementation manner of the first aspect, obtaining the second image based on the valid area of ​​the first image includes any one of the following:

[0045] cropping a valid area of ​​the first image to obtain the second image;

[0046] The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

[0047] The second image is obtained by processing the effective area of ​​the first image, so as to reduce the interference of image content outside the effective area in the first image on subsequent image processing.

[0048] In a second aspect, a data processing method is provided, the method comprising:

[0049] Receiving a first image sent by a computer device, wherein the first image is an image of a screen design diagram;

[0050] Recognize the first image to obtain boundary information of a module, wherein the module includes at least one visual component, and the boundary information includes a module position;

[0051] Identifying the visual components in the module to obtain the categories of the visual components;

[0052] The boundary information of the module and the category of the visual component are sent to the computer device.

[0053] The computer device acquires a first image, identifies the first image, obtains the boundary information of the module, identifies the visual components in the module, obtains the category of the visual components, creates visual components at the position corresponding to the boundary information of the module in the canvas according to the category of the visual components in the module, and thus obtains the screen layout of the target screen. The above process can directly identify the image containing the design draft, thereby obtaining the module division in the image and the category of the visual components in the module, and automatically generates the screen layout based on the recognition result, without the need for manual operation by the user, thereby reducing the waste of time cost.

[0054] In a first possible implementation manner of the second aspect, the identifying the first image to obtain the boundary information of the module includes:

[0055] Binarizing the first image to obtain a binary image;

[0056] Performing a dilation operation on black pixels in the binary image;

[0057] A black outline in the binary image after the dilation operation is extracted, and a position of the black outline that meets a preset outline condition in the first image is used as boundary information of the module.

[0058] The disclosed embodiment provides a method for identifying a module in a first image to obtain the boundary information of the module, which is beneficial for automatically generating visual components in the module according to the position of the module's boundary information on the canvas, thereby obtaining the screen layout of the target screen.

[0059] In a second possible implementation manner of the second aspect, the black outline that meets the preset outline conditions means: the area of ​​the black outline meets the preset area conditions, the black outline is the outermost outline among multiple nested outlines, and the shape of the black outline is a quadrilateral.

[0060] By limiting the black contour, it is ensured that the extracted black contour is a black contour that meets the preset contour conditions, so as to reduce the waste of time cost in the black contour extraction process.

[0061] In a third possible implementation manner of the second aspect, before using the position of the black outline that meets the preset outline condition in the first image as the boundary information of the module, the method further includes:

[0062] Normalizing the module to obtain boundary information of the normalized module;

[0063] The sending the boundary information of the module and the category of the visual component to the computer device comprises:

[0064] The normalized module boundary information and the category of the visual component are sent to the computer device.

[0065] By normalizing the modules in the first image, more accurate module boundary information can be obtained, thereby saving time for processing the modules and speeding up the generation of the picture layout.

[0066] In a fourth possible implementation manner of the second aspect, the normalizing the module includes at least one of the following:

[0067] Adjusting the boundary of the module to a regular quadrilateral;

[0068] Stretching the boundary of the module so that the module fills the canvas;

[0069] When there is an overlapping area between any two modules, the overlapping area between the any two modules is eliminated.

[0070] By standardizing the modules, the visual components in the subsequent modules can be displayed better on the canvas, thereby obtaining a better target screen layout.

[0071] In a fifth possible implementation manner of the second aspect, identifying the visual component in the module to obtain the category of the visual component includes:

[0072] The image block corresponding to the module in the first image is input into a visual component classification model, and the image block is identified by the visual component classification model to obtain the category of the visual component in the module.

[0073] The visual component classification model recognizes the image block corresponding to the module in the first image to obtain the category of the visual component in the module, thereby ensuring that the category of the visual component displayed in the subsequent screen layout is correct.

[0074] In a sixth possible implementation manner of the second aspect, after the image block is identified by the visual component classification model to obtain the category of the visual component in the module, the method further includes:

[0075] Receive feedback results from the computer device on the categories of the visual components in the module, and update the visual component classification model according to the feedback results.

[0076] When a feedback result on the category of the visual component in the module is received, the feedback result is sent to the server.

[0077] The server updates the visual component classification model based on the feedback result of the category of the visual component to improve the accuracy of the visual component classification model in classifying the visual component.

[0078] In a seventh possible implementation manner of the second aspect, the identifying the first image to obtain the boundary information of the module includes:

[0079] Based on the valid area of ​​the first image, obtain a second image, where the second image includes image content within the valid area;

[0080] The second image is recognized to obtain boundary information of the module, where the boundary information includes the module position.

[0081] When the first image is an image that includes areas outside the effective area, the effective area of ​​the first image needs to be identified to obtain the second image, so as to improve the recognition accuracy of the modules in the image.

[0082] In an eighth possible implementation manner of the second aspect, before obtaining the second image based on the valid area of ​​the first image, the method further includes any one of the following:

[0083] Based on the pixel value range corresponding to the white pixel points, determine the target contour in the first image, the target contour is the largest contour among the contours composed of the white pixel points in the first image, and take the area within the target contour as the valid area;

[0084] Perform edge detection on the first image to obtain at least one region, and take a region in the at least one region whose region shape meets the target condition as the valid region;

[0085] The first image is binarized to obtain at least one white area, and a white area in the at least one white area whose area shape meets the target condition is used as the effective area.

[0086] Based on the above three effective area recognition methods, the effective area recognition process of all images including those outside the effective area can be realized, and then the modules in the effective area can be recognized to obtain more accurate module boundary information.

[0087] In a ninth possible implementation manner of the second aspect, obtaining the second image based on the valid area of ​​the first image includes any one of the following:

[0088] cropping a valid area of ​​the first image to obtain the second image;

[0089] The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

[0090] The second image is obtained by processing the effective area of ​​the first image, so as to reduce the interference of image content outside the effective area in the first image on subsequent image processing.

[0091] In a third aspect, a data processing device is provided, which is used to execute the above data processing method. Specifically, the data processing device includes a functional module for executing the data processing method provided by the above first aspect or any optional manner of the above first aspect.

[0092] In a fourth aspect, a data processing device is provided, which is used to execute the above data processing method. Specifically, the data processing device includes a functional module for executing the data processing method provided by the above second aspect or any optional manner of the above second aspect.

[0093] In a fifth aspect, a computer device is provided, which includes one or more processors and one or more memories, wherein the one or more memories store at least one instruction, and the instruction is loaded and executed by the one or more processors to implement the operations performed by the above-mentioned data processing method.

[0094] In a sixth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the operations performed by the above-mentioned data processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 It is a schematic diagram of a specific implementation environment of a data processing method provided by an embodiment of the present disclosure;

[0096] Figure 2 is a block diagram of a computer device 200 according to an exemplary embodiment;

[0097] Figure 3 is a system architecture diagram according to an exemplary embodiment;

[0098] Figure 4 is an overall flow chart of a data processing method according to an exemplary embodiment;

[0099] Figure 5 is an interactive flow chart of a data processing method according to an exemplary embodiment;

[0100] Figure 6 is a data flow diagram of a data processing method according to an exemplary embodiment;

[0101] Figure 7 is a comparison diagram of an original first image and a second image according to an exemplary embodiment;

[0102] Figure 8 is a schematic diagram showing module boundary adjustment according to an exemplary embodiment;

[0103] Fig. 9 is a schematic diagram showing module boundary stretching according to an exemplary embodiment;

[0104] Fig.10 is a schematic diagram of module coverage according to an exemplary embodiment;

[0105] Fig.11 It is a process of training the visual component classification model provided by the embodiment of the present disclosure and a process of identifying the visual component category;

[0106] Fig.12 It is a partial example of the visual component drawn provided by the embodiment of the present disclosure;

[0107] Fig.13 is a feedback schematic diagram of a visual component provided by an embodiment of the present disclosure;

[0108] Fig.14A comparison diagram of screen layout generated based on a screen design diagram provided by an embodiment of the present disclosure;

[0109] Fig.15 is an interactive flow chart of a data processing method according to an exemplary embodiment;

[0110] Fig.16 is a diagram of a data processing device provided by an embodiment of the present disclosure;

[0111] Fig.17 It is a diagram of a data processing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0112] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0113] Figure 1 This is a schematic diagram of a specific implementation environment of a data processing method provided by an embodiment of the present disclosure, see Figure 1 The specific implementation environment of the data processing method includes: a server cluster 101 and multiple computer devices 102.

[0114] The server cluster 101 may include at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server cluster 101 is used to provide background services for applications that support data processing. For example, the server cluster 101 may be used to provide at least one of the functions of identifying the location and category of a module in an image during data processing, and send the identification result to the computer device 102 so that the computer device 102 can display the result based on the identification result.

[0115] In one possible implementation, the location and / or category recognition function may be performed by a trained recognition model, and accordingly, the server cluster 101 may include a server 1011 for recognition and a server 1012 for model training. Of course, the two servers may also be implemented on the same set of hardware, which is not limited in this regard in the embodiments of the present disclosure.

[0116] The computer device 102 is connected to the server cluster 101 via a wireless network or a wired network. The computer device 102 can be at least one of a smart phone, a desktop computer, a tablet computer, and a laptop computer. The computer device 102 can be used as an image provider, and the user can import the image on the computer device 102, and then send the image to the server cluster 101 for recognition, and then display it based on the recognition result returned by the server cluster 101. Of course, the computer device 102 can also independently serve as an image provider and an image recognizer, that is, the user can import the image on the computer device 102, and then the computer device 102 recognizes the image, and then displays it based on the recognition result, without the need to interact with the server cluster 101 in real time. The server cluster 101 can be used as a provider of the model used during recognition, and the computer device 102 can download the recognition model provided by the server cluster 101 at any time, so as to realize the above-mentioned image recognition based on the recognition model, and finally display it based on the recognition result.

[0117] The computer device 102 may generally refer to one of a plurality of computer devices. This embodiment is only illustrated by taking the computer device 102 as an example. Those skilled in the art may know that the number of the above-mentioned computer devices may be more or less. The embodiment of the present disclosure does not limit the number and type of computer devices.

[0118] Figure 2 2 is a block diagram of a computer device 200 according to an exemplary embodiment. For example, the computer device 200 may be provided as a user-side device or a server. Figure 2 , the computer device 200 includes a processing component 201, which further includes one or more processors, and a memory resource represented by a memory 202, for storing program codes, such as applications, that can be executed by the processing component 201. The application stored in the memory 202 may include one or more modules, each corresponding to a set of program codes. In addition, the processing component 202 is configured to execute the program code to perform the above-mentioned data processing method.

[0119] The computer device 200 may also include a power supply component 203 configured to perform power management of the computer device 200, a wired or wireless network interface 204 configured to connect the computer device 200 to a network, and an input / output (I / O) interface 205. The computer device 200 may operate based on an operating system stored in the memory 202, such as Windows Server 200. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TMor similar.

[0120] Next, based on the structural diagram of the above-mentioned computer device 200, Figure 3 The logical architecture of the overall system for implementing data processing is shown in FIG. 1 , and each function involved in the disclosed embodiment is briefly introduced. Figure 3 The system architecture can be divided into data layer, computing service layer, business layer and interaction layer. The data layer is used to provide various data required for data processing. These data can be stored in Figure 2 In the memory 202 shown, the computing service layer is used to provide model training and image processing functions. The business layer is used to provide various business-related functions, and the interaction layer is used to interact with users. The functions of the computing service layer, business layer, and interaction layer can be Figure 2 The processing component 201 shown is implemented. The following is a detailed introduction based on the functions of each layer:

[0121] The data layer is used to provide image data, business data, component configuration, and recognition result feedback. Among them, the image data is divided into two levels: the basic component library and the incremental component library. The basic component library is obtained through manual collection and collation, and is used to train the initial visual component classification model; the incremental component library is obtained through user feedback, specifically by the user reviewing the component prediction results and feedback the actual component category. The basic component library and the incremental component library together constitute the full component library. Business data can be used to associate business-related data for configuring visual components, such as power data of the power supply bureau, patient data of the hospital, etc. Business data can be the user's own business data. Component configuration can be achieved through the adaptation of the model-side output and the front-end orchestration unit input, and the orchestration function is used to parse the fields in the recognition results, select the corresponding visual components and visual component attributes, and display the development page results to the user. Recognition result feedback can be the feedback result of the user comparing the input and output visual components, which can be used to calculate the accuracy and recall rate.

[0122] The computing service layer is used to provide model services and image processing services, among which model services include offline training, online training, model distribution, and inference prediction of models. Offline training is based on the initial dataset through local training to obtain the first available component classification model; online training is that after the system collects user feedback results, it collects new data at the same time, and retrains the new model based on the previous version of the model through scheduled tasks; model distribution is to distribute the new model to each service node according to the deployment method; inference prediction is based on the hand-drawn module picture obtained by image processing, and uses the classification model to classify and predict the module. Image processing services also include effective area recognition, effective area cropping, module extraction, and layout normalization. Effective area recognition refers to determining the effective area of ​​the image after the user uploads the image. After the user confirms that the effective area is available, the area is cropped and perspective transformed to stretch the area back into a rectangle. Through edge detection, the user's hand-drawn module is extracted. Since the module edges are not neat, the module edges are aligned through layout normalization to make the image full and complete.

[0123] The business layer is used to provide image uploading, metadata parsing, page layout, component editing, template editing, data association, screen publishing, and user feedback. Among them, image uploading can be users uploading images through PC or mobile terminals; metadata parsing can be the layout unit parsing the image metadata transmitted by the computing service layer. The image metadata is composed of the visual component category metadata returned by the model service layer and the visual component basic metadata (coordinates / size and other image inherent information) returned by the image processing service. Page layout is to automatically layout the page according to the analysis results of the metadata; component editing provides the function of component editing, and template editing provides the function of template editing. Templates are some preset large-screen styles that come with the visualization system, and users can make adjustments based on these templates; data association is to associate business data in component configuration; screen publishing is to save and publish the large screen in development state online; user feedback is the feedback from users based on the layout results.

[0124] The interaction layer includes computer device upload, development editing, user feedback and release preview. Among them, terminal upload is the user uploading the visual interface through the PC or mobile terminal; development editing is the editing interface in the screen development process, and users can edit modules and visual components. User feedback is the user's feedback by comparing the screen development result map and the screen design map. Release preview is the preview interface generated during the screen development process, which is conducive to users viewing the entire development screen.

[0125] Below is a brief introduction based on the data processing process:

[0126] Figure 4 is an overall flow chart of a data processing method according to an exemplary embodiment. Figure 4 , the user uploads the screen design drawing to the computer device, the computer device obtains the screen design drawing and sends it to the server, the server identifies the effective area in the image of the screen design drawing, and the identification can adopt any of the following three identification methods, namely, identification based on HSV mask, identification based on Canny edge detection algorithm, and identification based on thresholding. Through the above identification, after the effective area is identified, the image of the effective area can be cropped and stretched, and the module in the cropped and stretched image can be extracted. The module can also be subjected to layout normalization processing. The layout normalization processing includes relative coverage elimination, module boundary adjustment to a regular quadrilateral, absolute coverage elimination, module boundary stretching, etc. After layout normalization processing, a module with a regular quadrilateral shape, no overlap between each other and full of the canvas is obtained, and the boundary information of the module after layout normalization is output. The image blocks corresponding to each module are input into the visual component classification model to obtain the category of the visual component in the module, and the category of the visual component is output. The output module boundary information constitutes metadata with the output module boundary information, and the metadata is used for the subsequent display and generation of the screen layout.

[0127] Next, a possible implementation of the data processing method is described based on the process of generating the entire screen layout through interaction between the computer device and the server:

[0128] See also Figure 5 and Figure 6 ,Should Figure 5 is an interactive flow chart of a data processing method according to an exemplary embodiment. Figure 6 FIG. 1 is a data flow diagram of a data processing method according to an exemplary embodiment, which specifically includes the following steps:

[0129] 501. A computer device obtains a first image, where the first image is an image of a screen design drawing.

[0130] The screen design drawing may be a screen design drawing hand-drawn by a user, a prototype image of a screen design drawing, a complete image of a screen design drawing on the Internet, etc., and the embodiments of the present disclosure are not limited to this.

[0131] In step 501, the computer device may obtain a first image by photographing or scanning the physical object of the screen design drawing, and save the first image in the computer device. The computer device is installed with an application, which is used for automatically generating the target screen layout. The user imports the first image saved in the computer device into the application, so that the computer device obtains the first image. For example, for a screen design drawing hand-drawn by a user, it can be photographed or scanned first, and the electronic image obtained by photographing or scanning can be saved in the computer device. When generating the target screen layout, the image of the hand-drawn screen design drawing is imported into the application.

[0132] 502. The computer device sends a first image to a server.

[0133] In step 502, the computer device acquires the first image and may automatically send the first image to the server, or may send the first image to the server after detecting a confirmation sending instruction from the user, which is not limited in the embodiments of the present disclosure.

[0134] 503. After receiving the first image sent by the computer device, the server identifies a valid area of ​​the first image.

[0135] The effective area refers to the area including the design content. For example, if the first image is an image of a hand-drawn screen design drawing, the electronic image obtained by photographing or scanning may include areas other than the hand-drawn paper. Therefore, the paper area must be identified first to determine the paper area in the first image, which is also the effective area of ​​the first image.

[0136] The above-mentioned identification of the effective area of ​​the first image can be implemented in any of the following ways:

[0137] Method 1: Identify paper area based on HSV mask.

[0138] The first method may specifically include: determining a target contour in the first image based on a pixel value range corresponding to white pixels, the target contour being the largest contour among contours composed of white pixels in the first image, and using the area within the target contour as a valid area.

[0139] A white pixel refers to a pixel whose display effect is white. For example, a white pixel may be a pixel whose H value is any value, whose S value falls within [0,100], and whose V value falls within [130,255].

[0140] Among them, the step of "determining the target contour in the first image based on the pixel value range corresponding to the white pixel point" in the above-mentioned method 1 may include the following steps: converting the first image into a first matrix, each triplet in the first matrix is ​​used to represent the HSV value of a pixel point in the first image, extracting a second matrix and a third matrix from the first matrix, the second matrix includes the S value in each triplet, and the third matrix includes the V value in each triplet, constructing a first mask matrix and a second mask matrix, the pixel values ​​of the elements in the first mask matrix and the second mask matrix are first preset pixel values, determining the first element position in the first matrix whose S value is greater than the preset S value, setting the pixel value corresponding to the first element position in the first mask matrix to the second preset pixel value, determining the second element position in the second matrix whose V value is less than the preset V value, setting the pixel value corresponding to the second element position in the second mask matrix to the second preset pixel value, superimposing the set first mask matrix and the second mask matrix to obtain a single-channel image, performing contour extraction on the single-channel image to obtain at least one contour, each contour is a contour composed of white pixels, and the largest contour among the at least one contour is used as the target contour.

[0141] For example, a first image with a height of H and a width of W is represented by HSV to obtain a first matrix with a size of [H, W, 3], and the S matrix (that is, the second matrix) and the V matrix (the third matrix) in the first matrix are extracted to construct an S matrix (the first matrix) and a V matrix (the second matrix) with a size of [H, W, 1]. The pixel values ​​of the pixels in the S matrix and the V matrix are all 255 (that is, the first preset pixel value). The positions of the pixels with pixel values ​​exceeding 100 in the S matrix (that is, the first element position) are screened, and the S matrix is ​​converted into an HSV matrix. The position of the corresponding pixel point in the S-cured film matrix is ​​set to 0 (that is, the second preset pixel value), the position of the pixel point with a pixel value less than 130 in the V matrix is ​​screened (that is, the second element position), and the position of the corresponding pixel point in the V-cured film matrix is ​​set to 0 (that is, the second preset pixel value). The set S-cured film matrix and the V-cured film matrix are superimposed together to obtain a single-channel image, and the contour of the single-channel image is extracted to obtain at least one contour composed of white pixels. The largest contour among the contours composed of white pixels is the target contour, that is, the area of ​​the paper.

[0142] It should be noted that the recognition using the HSV mask method is different from the recognition based on RGB (R is red, G is green, and B is blue). Since the pixel points in each color channel in the RGB pixel representation fall on [0,255], when the RGB color channel is used to identify the paper area, since the first image is obtained by taking a photo or scanning, there may be differences in light, which makes it impossible to determine the specific range of the white pixel points. HSV is a method of representing images using three channels of hue, saturation, and brightness. H is hue, S is saturation, and V is brightness. HSV is a color space suitable for human visual characteristics. The S component is a component related to the characteristics of the material. The S component is less affected by light, but the S component is more affected by paper. Among them, the value ranges of H, S, and V fall respectively in [0,360], [0,255], and [0,255]. The recognition based on HSV is not affected by light, making the recognition of the paper area more accurate.

[0143] Method 2: Recognition of paper area based on Canny edge detection.

[0144] Among them, Canny edge detection refers to extracting edge information from an image. The area composed of edge information is the area with actual image content, that is, the valid area, and the area outside the area composed of edge information is the invalid area.

[0145] The second method specifically includes: performing edge detection on the first image to obtain at least one region, and taking a region in the at least one region whose region shape meets the target condition as the effective region.

[0146] The first image is gray-scaled to obtain a first gray-scale image; the first gray-scale image is Gaussian-filtered to filter out noise in the image, and the image after Gaussian filtering is subjected to Canny edge detection; and according to the edge detection result, a quadrilateral region is selected as a paper region.

[0147] Among them, the step of "performing Canny edge detection on the image after Gaussian filtering" in the above-mentioned method 2 may include the following steps: using a Gaussian filter to process the first image to obtain a smooth image of the first image, the process filters out the noise of the first image, may amplify the edge of the first image, and reduce the recognition of pseudo edges in the first image to a certain extent; calculating the gradient intensity and gradient direction of each pixel point in the first image, suppressing the non-maximum values ​​therein, thereby eliminating the stray response caused by edge detection; applying a double threshold to detect the true edge and pseudo edge of the first image, the edge composed of pixels with a gradient value higher than the first threshold is a true edge, and the edge composed of pixels with a gradient value lower than the second threshold is a pseudo edge, wherein the first threshold is greater than the second threshold, and the pixels with gradient values ​​between the first threshold and the second threshold are suppressed. Further, it can also be detected whether the pseudo edge is connected to the true edge. When the pseudo edge is connected to the true edge, the pseudo edge is a true edge, and when the pseudo edge is not connected to the true edge, the pseudo edge is a false edge; through the above-mentioned further detection process, the situation where some pseudo edges may also be true edges is avoided, thereby improving the accuracy of edge detection.

[0148] Method 3: Paper area recognition based on thresholding.

[0149] The third method may specifically include: performing binarization processing on the first image to obtain at least one white area, and taking the white area in the at least one white area whose area shape meets the target condition as the effective area.

[0150] The specific recognition method is: according to the result of grayscale processing of the first image, the grayscale value of each pixel in the first image is obtained, the grayscale values ​​of all pixels are sorted, the median of the grayscale value is obtained, the pixel corresponding to the median of the grayscale value is set to white, and the pixel corresponding to the remaining grayscale values ​​is set to black to obtain at least one white area, the at least one white area is screened, and the white area whose area shape meets the target conditions is taken as the valid area.

[0151] Wherein, in any of the above-mentioned methods for identifying a valid area, the area shape meeting the target condition means that the shape of the area is a quadrilateral.

[0152] It should be noted that the above three effective area recognition methods can effectively cover the recognition of effective areas of most screen design drawings. When recognizing the effective area of ​​the first image, any of the above three recognition methods can be selected, or they can be used in sequence. When the previous recognition method cannot recognize the effective area, the next recognition method is triggered to recognize the effective area. The embodiments of the present disclosure do not limit this.

[0153] In the embodiment of the present disclosure, after obtaining the valid area of ​​the first image by the above three methods, the user needs to confirm that the valid area is the correct valid area by operating on the computer device, which can be implemented in any of the following ways:

[0154] In one implementation, after the server determines the valid area of ​​the first image, it marks the first image, which may be a graphic box, and sends the first image containing the valid area mark to a computer device, which displays the marked first image. When the user determines that the marked valid area is accurate, a confirmation instruction may be triggered through a confirmation operation, and the computer device sends the confirmation instruction to the server. After receiving the confirmation instruction, the server may determine the identified valid area as the valid area of ​​the first image, so as to execute subsequent steps based on the valid area.

[0155] In one implementation, when the user determines that the marked valid area is inaccurate, the valid area can be determined by manually marking on a computer device. For example, the manual marking method can be: the user marks four points on the first image displayed by the computer device through any input device, and the quadrilateral formed by the lines connecting the four points is the valid area of ​​the first image. The computer device sends the calibrated point coordinates to the server, and the server receives the point coordinates and determines the valid area of ​​the first image based on the point coordinates.

[0156] Through the above-mentioned recognition of the valid area, the interference of invalid information in the first image can be eliminated, making the subsequent recognition more accurate. Especially for the scenario where the first image is an image of a hand-drawn design drawing, the influence of non-paper areas, such as the background area captured by the camera, can be eliminated, thereby greatly improving the accuracy of subsequent recognition.

[0157] In addition, it should be noted that the effective area identification step is an optional step. During the data processing process, the effective area identification may not be performed, but the identification process may be performed directly based on the acquired image. Of course, the selection of whether to execute this step may be further based on the judgment of the image. For example, when the server receives the first image, it may first determine the source category of the first image. For example, if the first image is a photographed image of a hand-drawn design drawing, the effective area identification process is executed; if the first image is a scanned image of a hand-drawn design drawing, there is no need to execute the effective area identification step.

[0158] 504. Based on the valid area of ​​the first image, obtain a second image, where the second image includes image content in the valid area.

[0159] In the embodiment of the present disclosure, after the server identifies the valid area of ​​the first image, based on the valid area of ​​the first image, a specific method for acquiring the second image may be implemented in any of the following ways:

[0160] Implementation method 1: cropping the effective area of ​​the first image to obtain the second image.

[0161] After the server receives the instruction to confirm cropping sent by the computer device or the server receives the first image after calibrating four points sent by the computer device, it can directly crop the effective area of ​​the first image to obtain an image of the effective area, which is the second image.

[0162] Implementation method 2: cropping the effective area of ​​the first image to obtain an effective area image, and stretching the effective area image to obtain a second image.

[0163] After the server receives the instruction to confirm cropping sent by the computer device or the server receives the first image after four calibration points sent by the computer device, the server crops the effective area of ​​the first image to obtain an image of the effective area. The image of the effective area may not be a regular quadrilateral, so it needs to be stretched to obtain the stretched image of the effective area, which is the second image.

[0164] The stretching processing of the image of the above-mentioned effective area is also the perspective transformation of the image of the effective area. The perspective transformation refers to the transformation that utilizes the condition that the perspective center, image point and target point are collinear, and rotates the receiving surface (perspective surface) around the trace (perspective axis) by a certain angle according to the law of perspective rotation, thereby destroying the original projection light beam, while still keeping the projected geometric figure on the receiving surface unchanged.

[0165] It should be noted that the contents of the above steps 503 to 504 are morphological transformations and operations on the image. The above operations are beneficial to the subsequent extraction of modules in the image.

[0166] Of course, similar to the above step 503, step 504 is also an optional step, that is, for the disclosed embodiment, after the above step 502, when the server receives the first image, it can directly identify the first image to obtain the boundary information of the module without executing the image processing process of steps 503 and 504.

[0167] For example, see Figure 7 As shown, Figure 7 The upper figure is a display example of the original first image, and the lower figure is a second image obtained after the effective area is identified and processed such as cropping and stretching. The second image includes the image content in the effective area.

[0168] 505. The server identifies the second image and obtains boundary information of a module, where the module includes at least one visual component, and the boundary information includes a module position.

[0169] In a possible implementation, step 505 may specifically include the following steps:

[0170] 505A. Binarize the second image to obtain a binary image.

[0171] Before binarization, the second image can be grayed out first, and then the noise of the image background color can be reduced by blurring to obtain a second grayscale image. The second grayscale image can be binarized to obtain a binary image, which is an image composed of a white background and black lines.

[0172] 505B. Perform a dilation operation on the black pixels in the binary image.

[0173] The black pixels in the binary image are expanded by a dilation operation to eliminate the interference of white pixels in the black lines.

[0174] 505C. Extract the black contour in the binary image after the dilation operation, and use the position of the black contour that meets the preset contour condition in the second image as the boundary information of the module.

[0175] When multiple black contours are extracted, one black contour can be selected as the contour of the module, so as to determine the boundary information of the module based on the position of the selected black contour in the image. The black contour that meets the preset conditions means that: the area of ​​the black contour meets the preset area condition, the black contour is the outermost contour among multiple nested contours, and the shape of the black contour is a quadrilateral.

[0176] During the extraction process, each time a black contour is extracted, its area can be detected, and contours whose areas do not meet the preset area conditions can be screened out, for example, contours that are smaller than 3% of the second image can be screened out to obtain contours with larger areas, and subsequent selections can be made based on the multiple black contours finally obtained to obtain more accurate contours. It should be noted that during the contour extraction and conditional contour selection processes, screening can be performed based on the above-mentioned preset area conditions to achieve the purpose of more accurate detection and ensure that the obtained contours do contain the image content of the screen design drawing.

[0177] In step 505, the module may be normalized to obtain the boundary information of the normalized module to ensure that the obtained module has a regular shape and is suitable for screen display. The step of normalizing the module may specifically include any of the following:

[0178] (1) Adjust the boundary of the module to a regular quadrilateral.

[0179] The adjustment may be to adjust the height and width of the module to obtain a module with a regular quadrilateral shape.

[0180] by Figure 8 For example, the adjustment process Figure 8 The height of the module is adjusted as shown. The number of melanin in the module is detected from left to right. When the number of melanin on the right is greater than that on the left, the detection continues to the right. When the number of melanin on the right is less than that on the left, the height of the module is adjusted. Figure 8 As shown, when the number of melanin on the right side is less than that on the left side for the first time, the heights of modules A, B, and D are adjusted according to a certain ratio, and the adjusted heights are marked as adjusted. When the number of melanin on the right side is less than that on the left side for the second time, module B has been adjusted, so the detection continues to the right. When the number of melanin on the right side is less than that on the left side for the third time, module A has been adjusted, and the heights of modules C and E are adjusted according to a certain ratio below module A. The adjustment of the width of the module is similar to the adjustment of the height of the module, and will not be repeated here.

[0181] (2) Stretch the border of the module so that the module fills the canvas.

[0182] In step 505, the module boundary is stretched, and the specific steps for the stretched module to fill the canvas are:

[0183] According to the order of the module coordinate information from small to large, the boundaries of each module are stretched in turn, and the stretching order is upward, left, downward, and right, until it reaches the boundary of other adjacent modules or the boundary of the entire image, such as Fig. 9 Shown is a schematic diagram of module boundary extrusion.

[0184] (3) When there is an overlapping area between any two modules, the overlapping area between the two modules is eliminated.

[0185] In order to ensure that the modules are suitable for the screen display, it is possible to determine whether there is overlap between the modules according to their positions in the image. If there is overlap, the positions of the modules are adjusted until there is no overlap between the modules after adjustment. The specific method is as follows:

[0186] When there is an overlapping area between any two modules and the coverage is relative, that is, there is an overlapping area on the X-axis or Y-axis between the two modules, and the overlapping length is less than 1 / 3 of their respective widths, such as Fig.10Module B and module E in the second image have overlapping areas on the X axis. Get the vertex coordinates 1 and 2 of module B, get the vertex coordinates 5 and 6 of module E, and get the relative coverage length L, L = X1-X3. Move module B to the left by a certain distance, and move module E to the right by a certain distance, which is L / 2, so that there is no overlapping area between module B and module E on the X axis. Perform a relative coverage test on each module in the second image to ensure that there is no relative coverage between any two modules.

[0187] When there is an overlapping area between any two modules and the coverage is absolute, that is, there are common pixels between the two modules with absolute coverage, such as Fig.10 Module B and module C in the second image are obtained, vertex coordinates 1 and 2 of module B are obtained, vertex coordinates 3 and 4 of module C are obtained, and the coverage length L is obtained, L=X1-X2, module B is moved to the left by a certain length, and module C is moved to the right by a certain length, and the certain length is the coverage length L / 2, so that there are no common pixels between module B and module C. An absolute coverage detection is performed on each module in the second image to ensure that there is no absolute coverage between any two modules.

[0188] It should be noted that after any coverage removal operation is completed, the module boundary may be adjusted until it is adjusted to a regular quadrilateral. The specific operation is the same as the above-mentioned operation of adjusting the module boundary to a regular quadrilateral, and will not be repeated here.

[0189] Another point that needs to be explained is that the above process is described by taking the processing of the second image as an example. When steps 502 and 503 are not required, the processing of the first image can be carried out in the same way, which will not be described in detail here.

[0190] 506. The server identifies the visual component in the module and obtains the category of the visual component.

[0191] In an embodiment of the present disclosure, the visual component in the module is identified to obtain the category of the visual component, including the following steps: inputting the image block corresponding to the module in the second image into a visual component classification model, identifying the image block through the visual component classification model, and obtaining the category of the visual component in the module.

[0192] Among them, the visual component classification model is trained in the server in advance, such as Fig.11 The figure shows the training process of the visual component classification model and the identification process of the visual component category. The user can draw multiple visual components of different categories in advance. The categories of the visual components can be bar charts, line charts, maps, pie charts, liquid level charts, etc. Fig.12Some examples of visual components drawn as shown. Data enhancement is performed on the multiple visual components of different categories. The data enhancement is mainly to reduce the overfitting phenomenon of the network. Overfitting refers to making the hypothesis overly strict in order to obtain a consistent hypothesis. Avoiding overfitting is a core task in classifier design. A hypothesis can obtain a better fit than other hypotheses on the training data, but it cannot fit the data well on the data set outside the training data. At this time, it is considered that the hypothesis has overfitting. The data enhancement method includes: rotation, affine transformation, perspective transformation, corrosion and expansion, HSV perturbation, gamma perturbation, etc. After data enhancement, the data set is divided into a test set and a training set according to a 1:3 comparison, and stored in the file system. The visual components after data enhancement and the pre-drawn visual components are composed of image data, and the image data is trained offline to obtain a visual component classification model. The image of the screen design drawing is sent to a computer device, the image is processed, and the image blocks corresponding to each module in the image are obtained. The visual components in the image block are inferred and predicted by the visual component classification model to obtain the category of the visual component, and the category result is encapsulated. The user feedbacks whether the category is correct, and the feedback result and the picture data together constitute the full image data. The visual component classification model is trained online to obtain a visual component classification model with better generalization ability. Among them, the visual component classification model is a residual network ResNet50, and there are many models for image classification, such as DarkNet, VGG family models, inception family models, etc. can also replace the ResNet50. The image block corresponding to the module in the second image is input into the visual component classification model, and the image block is identified by the trained visual component classification model to obtain the category of the visual component in the module.

[0193] It should be noted that see Figure 6 The server may include a recognition server and a GPU server, wherein the GPU server may be used to train and update the visual component classification model, thereby sending the visual component classification model to the recognition server for use.

[0194] 507. The server sends the boundary information of the module and the category of the visual component to the computer device.

[0195] The server can compose metadata of the screen layout based on the obtained module boundary information and the category of the visual component. The metadata may include the component category, module position, module size, and the size of the overall image. The metadata may be in JSON format. For example, taking a pie chart as an example, the metadata and field meanings are as follows:

[0196] res_message: return message

[0197] res_code: return code

[0198] result: result body

[0199] module: module body

[0200] module_num: the number of modules parsed

[0201] photo_w: image width

[0202] photo_h: image height

[0203] pie_m: component category, here is a multi-pie chart

[0204] x: module starting x coordinate

[0205] y: module starting y coordinate

[0206] w: module width

[0207] h: Module height

[0208] x_percent: The starting x coordinate percentage of the module

[0209] y_percent: The starting y coordinate percentage of the module

[0210] w_percent: module width percentage

[0211] h_percent: Module height percentage

[0212] ind: module id

[0213] score: classification result accuracy score

[0214] 508. The computer device receives the boundary information of the module and the category of the visual component, creates a visual component at a position corresponding to the boundary information of the module in the canvas according to the category of the visual component in the module, and obtains a screen layout of the target screen.

[0215] In step 508, the computer device transmits the received metadata to the orchestration engine, which parses the metadata and arranges the layout of the entire large screen, the location of the modules and the categories of the visual components according to the parsed results of the metadata to obtain the screen layout of the target screen.

[0216] In the embodiment of the present disclosure, the computer device receives the boundary information of the module and the category of the visual component sent by the server, and based on the visual component classification model in the server, provides feedback on whether the result of the classification of the visual component in the second image is correct, and sends the feedback result to the server, wherein the feedback result includes but is not limited to the category of the visual component, the image block corresponding to the visual component, the index value of the predicted category of the visual component, and the accuracy rate of the predicted category of the visual component, such as Fig.13 The figure shows a feedback diagram of a visual component. The server updates the visual component classification model according to the feedback results of the computer device. Whenever the updated visual component classification model is obtained, the server distributes the model to each prediction server to ensure the reliability of the visual component classification model. The server can encapsulate the feedback results of the visual component sent by the computer device into an incremental visual component library, which together with the basic visual component library constitutes a full visual component library, and the full visual component library is used for the subsequent visual component classification model to identify the category of the visual component in the module.

[0217] It should be noted that the canvas may be created before the computer device sends the first image to the server, or after the computer device receives the module boundary information and the category of the visual component, which is not limited in the embodiments of the present disclosure.

[0218] 509. The computer device displays the screen layout of the target screen.

[0219] In step 509, the target screen layout is displayed on the computer device. Further, the computer device is configured with business data, which can be associated with the business data and displayed in the target screen layout, wherein the business data refers to the data that the user has, such as the power data of the power supply bureau, the patient data of the hospital, etc.

[0220] In the embodiment of the present disclosure, after the computer device displays the screen layout of the target screen, Fig.14 The figure shows a comparison chart of screen layouts generated based on a screen design drawing. If a computer device user finds that the screen layout is inconsistent with the expected implementation after viewing the screen layout, the screen layout can be adjusted based on the screen design drawing and the adjusted screen layout can be displayed. If the adjusted screen layout is still inconsistent with the expected implementation, the screen design drawing can be redesigned.

[0221] The method provided by the embodiment of the present disclosure is that a computer device sends a first image to a server, the server identifies the first image to obtain boundary information of a module in the first image, the boundary information includes the position of the module, the server identifies the visual component in the module to obtain the category of the visual component, and sends the boundary information of the module and the category of the visual component to the computer device. The above process can directly identify the image containing the design draft, so as to obtain the module division in the image and the category of the visual component in the module, and automatically generate the screen layout based on the recognition result, without the need for manual operation by the user, thereby reducing the waste of time cost.

[0222] See also Fig.15 ,Should Fig.15 is an interactive flow chart of a data processing method according to an exemplary embodiment, which is different from the above Figure 5 In the embodiment shown, the computer device and the server interact to complete the generation process of the entire screen layout. Fig.15 The corresponding data processing method is that the computer device completes the image acquisition, recognition and subsequent display processes alone, thereby completing the screen layout generation process, which specifically includes the following steps:

[0223] 1501. A computer device obtains a first image, where the first image is an image of a screen design drawing.

[0224] In the case where the computer device has image recognition capability, the computer device may directly execute subsequent step 1502 after acquiring the first image without sending the first image to the server.

[0225] 1502. The computer device identifies a valid area of ​​the first image.

[0226] 1503. The computer device obtains a second image based on the valid area of ​​the first image, where the second image includes image content in the valid area.

[0227] 1504. The computer device recognizes the second image to obtain boundary information of a module, where the module includes at least one visual component, and the boundary information includes a module position.

[0228] 1505. The computer device identifies the visual components in the module and obtains the categories of the visual components in the module.

[0229] The above steps 1502 to 1505 are similar to the above steps 503 to 506, and are not described in detail here.

[0230] In step 1505, the computer device identifies the visual component in the module, and after obtaining the category of the visual component, provides feedback on whether the classification result of the visual component is correct based on the visual component classification model, and sends the feedback result to the visual component classification model. The feedback result includes but is not limited to the category of the visual component, the image block corresponding to the visual component, the index value of the visual component category prediction, and the accuracy of the visual component category prediction. The visual component classification model updates the visual component classification model based on the feedback result to ensure the reliability of the visual component classification model.

[0231] 1506. The computer device creates a visual component according to the module boundary information and the category of the visual component at a position corresponding to the module boundary information in the canvas and according to the category of the visual component in the module to obtain a screen layout of the target screen.

[0232] In step 1506, the server does not need to participate in the generation process of the target screen's picture layout. The computer device can directly create the visual component according to the category of the visual component at the position corresponding to the module's boundary information in the canvas based on the module's boundary information and the category of the visual component, thereby obtaining the target screen's picture layout.

[0233] 1507. The computer device displays the screen layout of the target screen.

[0234] The method provided by the embodiment of the present disclosure is that a computer device acquires a first image, identifies the first image, obtains the boundary information of the module, identifies the visual components in the module, obtains the category of the visual components, creates visual components according to the category of the visual components in the module at the position corresponding to the boundary information of the module in the canvas, and thus obtains the screen layout of the target screen. The above process can directly identify the image containing the design draft, thereby obtaining the module division in the image and the category of the visual components in the module, and automatically generates the screen layout based on the recognition result, without the need for manual operation by the user, thereby reducing the waste of time cost.

[0235] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0236] Fig.16 is a diagram of a data processing device provided by an embodiment of the present disclosure. Fig.16 , the device comprises:

[0237] The receiving unit 1601 is used to execute the above step 503. The receiving unit 1601 is also used to execute the above step 508;

[0238] A boundary identification unit 1602, configured to execute the above step 505;

[0239] Category identification unit 1603, used to execute the above step 506;

[0240] The sending unit 1604 is used to execute the above step 502. The sending unit 1604 is also used to execute the above step 507.

[0241] In a possible implementation, the boundary identification unit 1602 includes:

[0242] A binarization processing subunit, used to perform binarization processing on the first image in the above step 505 to obtain a binary image;

[0243] The expansion subunit is used to perform the expansion operation on the black pixels in the binary image in the above step 505;

[0244] The extraction subunit is used to extract the black outline in the binary image after the dilation operation in step 505.

[0245] In a possible implementation, the device further includes:

[0246] The normalization unit is used to perform normalization on the module in the above step 505 to obtain the boundary information of the normalized module.

[0247] In a possible implementation, the normalization unit includes one or more of the following subunits:

[0248] An adjusting subunit, used to execute the above step 505 to adjust the boundary of the module into a regular quadrilateral;

[0249] The stretching subunit is used to execute the stretching of the border of the module in the above step 505, so that the module after stretching fills the canvas;

[0250] The elimination subunit is used to execute the above step 505 to eliminate the coverage between any two modules when there is a coverage area between the any two modules.

[0251] In a possible implementation, the category identification unit 1603 is used to execute step 506 of inputting the image block corresponding to the module in the first image into a visual component classification model, identifying the image block through the visual component classification model, and obtaining the category of the visual component in the module.

[0252] In a possible implementation, the receiving unit 1601 is further configured to execute step 508 of receiving feedback results from a computer device on categories of visual components in a module, and updating the visual component classification model according to the feedback results.

[0253] In a possible implementation, the boundary identification unit 1602 is configured to execute step 504 .

[0254] In a possible implementation, the device further includes:

[0255] An image processing unit, configured to perform any of the following items in step 504:

[0256] Cutting the effective area of ​​the first image to obtain a second image;

[0257] The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

[0258] The above-mentioned device sends a first image to a server by a computer device, and the server identifies the first image to obtain boundary information of the module in the first image, and the boundary information includes the position of the module. The server identifies the visual component in the module to obtain the category of the visual component, and sends the boundary information of the module and the category of the visual component to the computer device. The above-mentioned process can directly identify the image containing the design draft, thereby obtaining the module division in the image and the category of the visual component in the module, and automatically generates the screen layout based on the recognition result, without the need for manual operation by the user, thereby reducing the waste of time cost.

[0259] Fig.17 is a diagram of a data processing device provided by an embodiment of the present disclosure. Fig.17 , the device comprises:

[0260] An acquisition unit 1701 is used to execute the above step 1501. The acquisition unit 1701 is also used to execute the above step 1503;

[0261] A boundary identification unit 1702, configured to execute the above step 1504;

[0262] A category identification unit 1703, configured to execute the above step 1505;

[0263] The creation unit 1704 is used to execute the above step 1506.

[0264] In a possible implementation, the boundary identification unit 1702 includes:

[0265] A binarization processing subunit, used to perform binarization processing on the first image in the above step 1504 to obtain a binary image;

[0266] The expansion subunit is used to perform the expansion operation on the black pixels in the binary image in the above step 1504;

[0267] The extraction subunit is used to extract the black contour in the binary image after the dilation operation in the above step 1504, and use the position of the black contour that meets the preset contour condition in the first image as the boundary information of the module.

[0268] In a possible implementation, the device further includes:

[0269] The normalization unit is used to perform normalization of the module in the above step 1504 to obtain the boundary information of the normalized module.

[0270] In a possible implementation, the normalization unit includes any one or more of the following subunits:

[0271] An adjusting subunit, used to execute the above step 1504 to adjust the boundary of the module into a regular quadrilateral;

[0272] The stretching subunit is used to execute the stretching of the module boundary in the above step 1504, so that the module fills the canvas after stretching;

[0273] The elimination subunit is used to execute the above step 1504 to eliminate the coverage between any two modules when there is a coverage area between the any two modules.

[0274] In a possible implementation, the category identification unit 1703 is also used to execute step 1505 of inputting the image block corresponding to the module in the first image into the visual component classification model, identifying the image block through the visual component classification model, and obtaining the category of the visual component in the module.

[0275] In a possible implementation, the device further includes:

[0276] The sending unit is used to execute the above step 1505, when receiving the feedback result on the category of the visual component in the module, send the feedback result to the server.

[0277] In a possible implementation, the boundary identification unit 1702 is also used to execute step 1503 based on the valid area of ​​the first image to obtain a second image, wherein the second image includes the image content within the valid area; and identify the second image to obtain boundary information of the module, wherein the boundary information includes the module position.

[0278] In a possible implementation, the device further includes:

[0279] An image processing unit, configured to perform any of the following in step 1503:

[0280] Cutting the effective area of ​​the first image to obtain a second image;

[0281] The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

[0282] The above-mentioned device obtains a first image by a computer device, identifies the first image, obtains the boundary information of the module, identifies the visual components in the module, obtains the category of the visual components, creates the visual components according to the category of the visual components in the module at the position corresponding to the boundary information of the module in the canvas, and thus obtains the screen layout of the target screen. The above-mentioned process can directly identify the image containing the design draft, thereby obtaining the module division in the image and the category of the visual components in the module, and automatically generates the screen layout based on the recognition result, without the need for manual operation by the user, thereby reducing the waste of time cost.

[0283] It should be noted that: the data processing device provided in the above embodiment only uses the division of the above functional modules as an example for data processing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0284] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor to complete the data processing method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0285] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0286] The above are only optional embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a first image, where the first image is an image of a screen design diagram; Recognize the first image to obtain boundary information of a module, wherein the module includes at least one visual component, and the boundary information includes a module position; Identify the visual components in the module to obtain categories of the visual components, where the categories of the visual components include bar charts, line charts, maps, pie charts, text, pictures, or liquid level charts; At a position corresponding to the boundary information of the module in the canvas, the visual component is created according to the category of the visual component in the module to obtain a screen layout of the target screen.

2. The method according to claim 1, characterized in that The step of recognizing the first image to obtain boundary information of a module includes: Binarizing the first image to obtain a binary image; Performing a dilation operation on black pixels in the binary image; A black outline in the binary image after the dilation operation is extracted, and a position of the black outline that meets a preset outline condition in the first image is used as boundary information of the module.

3. The method according to claim 2, characterized in that The black contour meeting the preset contour condition means that: the area of ​​the black contour meets the preset area condition, the black contour is the outermost contour among multiple nested contours, and the shape of the black contour is a quadrilateral.

4. The method according to claim 2, characterized in that: After taking the position of the black outline meeting the preset outline condition in the first image as the boundary information of the module, the method further comprises: Normalizing the module to obtain boundary information of the normalized module; Creating the visual component at a position corresponding to the boundary information of the module in the canvas according to the category of the visual component in the module includes: At a position corresponding to the boundary information of the normalized module in the canvas, the visual component is created according to the category of the visual component in the module.

5. The method according to claim 4, characterized in that The normalizing of the module comprises at least one of the following: Adjusting the boundary of the module to a regular quadrilateral; Stretching the boundary of the module so that the module fills the canvas; When there is an overlapping area between any two modules, the overlapping area between the any two modules is eliminated.

6. The method according to claim 1, characterized in that The identifying the visual component in the module to obtain the category of the visual component includes: The image block corresponding to the module in the first image is input into a visual component classification model, and the image block is identified by the visual component classification model to obtain the category of the visual component in the module.

7. The method according to claim 6, characterized in that After the image block is identified by the visual component classification model to obtain the category of the visual component in the module, the method further includes: When a feedback result on the category of the visual component in the module is received, the feedback result is sent to the server.

8. The method according to claim 1, characterized in that: The step of recognizing the first image to obtain boundary information of a module includes: Based on the valid area of ​​the first image, obtain a second image, where the second image includes image content within the valid area; The second image is recognized to obtain boundary information of the module, where the boundary information includes the module position.

9. The method according to claim 8, characterized in that Before obtaining the second image based on the valid area of ​​the first image, the method further includes any one of the following: Based on the pixel value range corresponding to the white pixel points, determine the target contour in the first image, the target contour is the largest contour among the contours composed of the white pixel points in the first image, and take the area within the target contour as the valid area; Perform edge detection on the first image to obtain at least one region, and take a region in the at least one region whose region shape meets the target condition as the valid region; The first image is binarized to obtain at least one white area, and a white area in the at least one white area whose area shape meets the target condition is used as the effective area.

10. The method according to claim 8, characterized in that The obtaining of the second image based on the valid area of ​​the first image includes any one of the following: cropping a valid area of ​​the first image to obtain the second image; The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

11. A data processing method, characterized in that: The method comprises: Receiving a first image sent by a computer device, wherein the first image is an image of a screen design diagram; Recognize the first image to obtain boundary information of a module, wherein the module includes at least one visual component, and the boundary information includes a module position; Identify the visual components in the module to obtain categories of the visual components, where the categories of the visual components include bar charts, line charts, maps, pie charts, text, pictures, or liquid level charts; The boundary information of the module and the category of the visual component are sent to the computer device. The boundary information of the module and the category of the visual component are used by the computer device to create the visual component at the position corresponding to the boundary information of the module in the canvas according to the category of the visual component in the module to obtain the screen layout of the target screen.

12. The method according to claim 11, characterized in that The step of recognizing the first image to obtain boundary information of a module includes: Binarizing the first image to obtain a binary image; Performing a dilation operation on black pixels in the binary image; A black outline in the binary image after the dilation operation is extracted, and a position of the black outline that meets a preset outline condition in the first image is used as boundary information of the module.

13. The method according to claim 12, characterized in that The black contour meeting the preset contour condition means that: the area of ​​the black contour meets the preset area condition, the black contour is the outermost contour among multiple nested contours, and the shape of the black contour is a quadrilateral.

14. The method according to claim 12, characterized in that After taking the position of the black outline meeting the preset outline condition in the first image as the boundary information of the module, the method further comprises: Normalizing the module to obtain boundary information of the normalized module; The sending the boundary information of the module and the category of the visual component to the computer device comprises: The normalized module boundary information and the category of the visual component are sent to the computer device.

15. The method according to claim 14, characterized in that The normalizing of the module comprises at least one of the following: Adjusting the boundary of the module to a regular quadrilateral; Stretching the boundary of the module so that the module fills the canvas; When there is an overlapping area between any two modules, the overlapping area between the any two modules is eliminated.

16. The method according to claim 11, characterized in that The identifying the visual component in the module to obtain the category of the visual component includes: The image block corresponding to the module in the first image is input into a visual component classification model, and the image block is identified by the visual component classification model to obtain the category of the visual component in the module.

17. The method according to claim 16, characterized in that After the image block is identified by the visual component classification model to obtain the category of the visual component in the module, the method further includes: Receive feedback results from the computer device on the categories of the visual components in the module, and update the visual component classification model according to the feedback results.

18. The method according to claim 11, characterized in that The step of recognizing the first image to obtain boundary information of a module includes: Based on the valid area of ​​the first image, obtain a second image, where the second image includes image content within the valid area; The second image is recognized to obtain boundary information of the module, where the boundary information includes the module position.

19. The method according to claim 18, characterized in that Before obtaining the second image based on the valid area of ​​the first image, the method further includes any one of the following: Based on the pixel value range corresponding to the white pixel points, determine the target contour in the first image, the target contour is the largest contour among the contours composed of the white pixel points in the first image, and take the area within the target contour as the valid area; Perform edge detection on the first image to obtain at least one region, and take a region in the at least one region whose region shape meets the target condition as the valid region; The first image is binarized to obtain at least one white area, and a white area in the at least one white area whose area shape meets the target condition is used as the effective area.

20. The method according to claim 18, characterized in that The obtaining of the second image based on the valid area of ​​the first image includes any one of the following: cropping a valid area of ​​the first image to obtain the second image; The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

21. A data processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a first image, wherein the first image is an image of a screen design diagram; a boundary recognition unit, configured to recognize the first image to obtain boundary information of a module, wherein the module includes at least one visual component, and the boundary information includes a module position; A category identification unit, used for identifying the visual components in the module to obtain the category of the visual components, wherein the category of the visual components includes a bar chart, a line chart, a map, a pie chart, text, a picture or a liquid level chart; The creation unit is used to create the visual component at a position corresponding to the boundary information of the module in the canvas according to the category of the visual component in the module to obtain the screen layout of the target screen.

22. The device according to claim 21, characterized in that The boundary recognition unit comprises: A binarization processing subunit, used for performing binarization processing on the first image to obtain a binary image; An expansion subunit, used for performing an expansion operation on black pixels in the binary image; The extraction subunit is used to extract the black contour in the binary image after the dilation operation, and use the position of the black contour that meets the preset contour condition in the first image as the boundary information of the module.

23. The device according to claim 21, characterized in that The device also includes: A normalization unit, used to normalize the module to obtain boundary information of the normalized module; The creation unit is used to create the visual component according to the category of the visual component in the module at a position corresponding to the boundary information of the normalized module in the canvas.

24. The device according to claim 23, characterized in that The normalization unit includes any one or more of the following subunits: An adjusting subunit, used for adjusting the boundary of the module into a regular quadrilateral; A stretching subunit, used to stretch the border of the module so that the module fills the canvas after stretching; The elimination subunit is used to eliminate the coverage between any two modules when there is a coverage area between any two modules.

25. The device according to claim 21, characterized in that The category recognition unit is used to input the image block corresponding to the module in the first image into the visual component classification model, identify the image block through the visual component classification model, and obtain the category of the visual component in the module.

26. The device according to claim 21, characterized in that The device also includes: The sending unit is used to send the feedback result to the server when receiving the feedback result on the category of the visual component in the module.

27. The device according to claim 21, characterized in that The boundary recognition unit is used to obtain a second image based on the effective area of ​​the first image, and the second image includes the image content in the effective area; recognize the second image to obtain the boundary information of the module, and the boundary information includes the module position.

28. The device according to claim 27, characterized in that The apparatus further includes: an image processing unit, configured to perform any of the following: cropping a valid area of ​​the first image to obtain the second image; The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

29. A data processing device, characterized in that: The device comprises: A receiving unit, configured to receive a first image sent by a computer device, wherein the first image is an image of a screen design diagram; a boundary recognition unit, configured to recognize the first image to obtain boundary information of a module, wherein the module includes at least one visual component, and the boundary information includes a module position; A category identification unit, used for identifying the visual components in the module to obtain the category of the visual components, wherein the category of the visual components includes a bar chart, a line chart, a map, a pie chart, text, a picture or a liquid level chart; A sending unit is used to send the boundary information of the module and the category of the visual component to the computer device. The boundary information of the module and the category of the visual component are used by the computer device to create the visual component according to the category of the visual component in the module at a position corresponding to the boundary information of the module in the canvas, so as to obtain the screen layout of the target screen.

30. The device according to claim 29, characterized in that The boundary recognition unit comprises: A binarization processing subunit, used for performing binarization processing on the first image to obtain a binary image; An expansion subunit, used for performing an expansion operation on black pixels in the binary image; The extraction subunit is used to extract the black contour in the binary image after the dilation operation, and use the position of the black contour that meets the preset contour condition in the first image as the boundary information of the module.

31. The device according to claim 29, characterized in that The device also includes: A normalization unit, used to normalize the module to obtain boundary information of the normalized module; The sending unit is used to send the normalized module boundary information and the category of the visual component to the computer device.

32. The device according to claim 31, characterized in that The normalization unit includes one or more of the following subunits: An adjusting subunit, used for adjusting the boundary of the module into a regular quadrilateral; A stretching subunit, used to stretch the border of the module so that the module fills the canvas after stretching; The elimination subunit is used to eliminate the coverage between any two modules when there is a coverage area between any two modules.

33. The device according to claim 29, characterized in that The category recognition unit is used to input the image block corresponding to the module in the first image into the visual component classification model, identify the image block through the visual component classification model, and obtain the category of the visual component in the module.

34. The device according to claim 29, characterized in that The receiving unit is further configured to receive feedback results from the computer device on the categories of the visual components in the module, and update the visual component classification model according to the feedback results.

35. The device according to claim 29, characterized in that The boundary recognition unit is used to obtain a second image based on the effective area of ​​the first image, and the second image includes the image content in the effective area; recognize the second image to obtain the boundary information of the module, and the boundary information includes the module position.

36. The device according to claim 35, characterized in that The apparatus further includes an image processing unit, configured to perform any of the following: cropping a valid area of ​​the first image to obtain the second image; The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

37. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one instruction is stored in the one or more memories, and the instruction is loaded and executed by the one or more processors to implement the operation performed by the data processing method of any one of claims 1 to 10 or claims 11 to 20.

38. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operation performed by the data processing method according to any one of claims 1 to 10 or claims 11 to 20.

Images