Data processing method, device, computer equipment, and computer-readable storage medium

Abstract

The present disclosure provides a data processing method, apparatus, computer device, and computer-readable storage medium, all belonging to the field of data visualization. The method comprises: transmitting a first image to a server via a computer device; the server identifying the first image to obtain the categories and locations of visual components in the first image; and transmitting the categories and locations of the visual components to the computer device. The above process can directly identify an image containing a design draft, thereby obtaining the division of a target area in the image and the categories of visual components in the target area. The screen layout is automatically generated based on the identification results, eliminating the need for manual user operation and thus reducing time and cost.

Description

Technical Field

[0001] The present disclosure relates to the field of data visualization, and in particular to a data processing method, apparatus, computer device, and computer-readable storage medium. Background Art

[0002] With the booming big data industry, more and more companies are realizing the importance of data management and application, particularly the use of data visualization technology to present data. Data visualization is the visualization of data in a database. It is the application of visualization technology to non-spatial data. It uses graphics or images to present data and its structural relationships in a more intuitive way. The basic concept is to treat individual data items in the database as individual primitives, construct data images from a large number of data sets, and simultaneously represent the various attribute values ​​of the data in a multi-dimensional manner, allowing for deeper observation and analysis of the data. Through data visualization, the process of understanding multi-dimensional data is reduced to a simple process of observing color, length, and height, significantly reducing the time required to understand data. As a result, more and more companies are using data visualization technology to display, monitor, and analyze various operational data.

[0003] At present, when a user obtains a screen design drawing with a good design concept, he hopes to make some adaptive modifications based on the screen design drawing. The user can create a canvas in the data visualization system, and refer to the visual components in the obtained screen design drawing. From the visual component library preset in the data visualization system, the user manually drags and drops the corresponding visual components to the canvas to generate the screen layout of the screen design drawing, and then makes some modifications to the screen layout to obtain a screen layout with better display effect, and displays data according to the modified screen layout.

[0004] During the above data processing, the entire component layout is completed by manually dragging and dropping. This process is cumbersome and easily results in a waste of time and cost. Summary of the Invention

[0005] The embodiments of the present disclosure provide a data processing method, apparatus, computer device, and computer-readable storage medium, which can solve the technical problem in related technologies of cumbersome operation steps and easy waste of time and cost. The technical solution is as follows:

[0006] In a first aspect, a data processing method is provided, the method comprising:

[0007] Acquire a first image, where the first image is an image of a screen design drawing;

[0008] Identifying visual components in the first image to obtain categories and positions of the visual components in the first image;

[0009] According to the category and position of the visual component, the visual component is created in the canvas to obtain the screen layout of the target screen.

[0010] A computer device sends a first image to a server, which identifies the first image to obtain the category and position of the visual component in the first image, and sends the category and position of the visual component to the computer device. The above process can directly identify the image containing the design draft, thereby obtaining the division of the target area in the image and the category of the visual component in the target area, and automatically generates the screen layout based on the identification results without the need for manual operation by the user, thereby reducing the waste of time costs.

[0011] In a first possible implementation manner of the first aspect, identifying the visual component in the first image to obtain the category and position of the visual component in the first image includes:

[0012] The first image is input into a visual component classification model, and target detection is performed on multiple candidate areas in the first image through the visual component classification model to obtain the category and position of the visual component.

[0013] The visual component classification model identifies the visual component in the first image to obtain the category and position of the visual component, thereby ensuring that the category and position of the visual component displayed in the subsequent screen layout are correct.

[0014] In a second possible implementation of the first aspect,

[0015] After performing target detection on multiple candidate areas in the first image using the visual component classification model to obtain the category and position of the visual component, the method further includes:

[0016] When a feedback result on the category of the visual component in the first image is received, the feedback result is sent to the server.

[0017] The server updates the visual component classification model based on the feedback result of the category of the visual component to improve the accuracy of the visual component classification model in classifying the visual component.

[0018] In a third possible implementation manner of the first aspect, the method further includes at least one of the following:

[0019] extracting the color of the visual component in the first image;

[0020] extracting the background color of the first image;

[0021] The step of creating the visual component in the canvas according to the category and position of the visual component to obtain the layout of the target screen includes:

[0022] The visual component is created in a canvas according to the extracted color and the category and position of the visual component to obtain a screen layout of the target screen.

[0023] By extracting the background color of the first image and the color of the visual component in the first image, the color attribute of the visual component is obtained. The subsequent target screen can be a target screen with color rendering, and the color rendering is automatically rendered, reducing the waste of time cost.

[0024] In a fourth possible implementation manner of the first aspect, the method further includes:

[0025] Normalizing the position of the visual component;

[0026] The step of creating the visual component in the canvas according to the category and position of the visual component to obtain the layout of the target screen includes:

[0027] According to the category and normalized position of the visual component, the visual component is created in the canvas to obtain the screen layout of the target screen.

[0028] By normalizing the positions of the visual components in the first image, more accurate position information of the visual components can be obtained, thereby saving time for processing the visual components and accelerating the generation of the picture layout.

[0029] In a fifth possible implementation manner of the first aspect, normalizing the position of the visual component includes at least one of the following:

[0030] Adjusting the border of the visual component to a regular quadrilateral;

[0031] Stretching the boundary of the visual component so that the stretched visual component fills the canvas;

[0032] When there is an overlapping area between any two visual components, the overlapping between the any two visual components is eliminated.

[0033] By normalizing the visual component so that its boundary is a regular quadrilateral, filling the entire canvas and without overlapping modules, subsequent visual components can be displayed better on the canvas, thereby obtaining a better target screen layout.

[0034] In a sixth possible implementation manner of the first aspect, the method further includes:

[0035] Perform noise reduction processing on the first image.

[0036] Through the noise reduction process, the interference of the image resolution in the first image is eliminated, so that the recognition process of the visual component in the first image is more accurate.

[0037] In a seventh possible implementation manner of the first aspect, recognizing the first image to obtain the category and position of the visual component includes:

[0038] obtaining a second image based on a valid area of ​​the first image, wherein the second image includes image content within the valid area;

[0039] The second image is identified to obtain the category and position of the visual component.

[0040] When the original image includes an image outside the effective area, the effective area of ​​the original image needs to be identified to obtain the first image, so as to improve the recognition accuracy of the visual components in the image.

[0041] In an eighth possible implementation manner of the first aspect, obtaining the second image based on the valid area of ​​the first image includes any one of the following:

[0042] cropping a valid area of ​​the first image to obtain the second image;

[0043] The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

[0044] The first image is obtained by processing the effective area of ​​the original image, so as to reduce the interference of image content outside the effective area in the original image on subsequent image processing.

[0045] In a second aspect, a data processing method is provided, the method comprising:

[0046] receiving a first image sent by a computer device, wherein the first image is an image of a screen design;

[0047] Identifying visual components in the first image to obtain categories and positions of the visual components in the first image;

[0048] The category and location of the visual component are sent to the computer device.

[0049] A computer device captures a first image, identifies visual components in the first image, obtains the category and position of the visual components, and creates the visual components on a canvas based on the category and position of the visual components, thereby obtaining a screen layout for the target screen. This process can directly identify an image containing a design drawing, thereby obtaining the location and category of the visual components in the image, and generating a screen layout based on the identification results, without requiring manual user operation, thereby reducing time and cost.

[0050] In a first possible implementation manner of the second aspect, identifying the visual component in the first image to obtain the category and position of the visual component in the first image includes:

[0051] The first image is input into a visual component classification model, and target detection is performed on multiple candidate areas in the first image through the visual component classification model to obtain the category and position of the visual component.

[0052] The visual component classification model identifies the visual component in the first image to obtain the category and position of the visual component, thereby ensuring that the category and position of the visual component displayed in the subsequent screen layout are correct.

[0053] In a second possible implementation manner of the second aspect, after performing target detection on multiple candidate regions in the first image using the visual component classification model to obtain the category and position of the visual component, the method further includes:

[0054] When receiving a feedback result from the computer device on the category of the visual component in the first image, the visual component classification model is updated according to the feedback result.

[0055] The server updates the visual component classification model based on the feedback result of the category of the visual component to improve the accuracy of the visual component classification model in classifying the visual component.

[0056] In a third possible implementation manner of the second aspect, the method further includes at least one of the following:

[0057] extracting the color of the visual component in the first image;

[0058] extracting the background color of the first image;

[0059] The sending the category and position of the visual component to the computer device includes:

[0060] The extracted color and the category and position of the visual component are sent to the computer device.

[0061] By extracting the background color of the first image and the color of the visual component in the first image, the color attribute of the visual component is obtained. The subsequent target screen can be a target screen with color rendering, and the color rendering is automatically rendered, reducing the waste of time cost.

[0062] In a fourth possible implementation manner of the second aspect, the method further includes:

[0063] Normalizing the position of the visual component;

[0064] The sending the category and position of the visual component to the computer device includes:

[0065] The normalized category and position of the visual component are sent to the computer device.

[0066] By normalizing the positions of the visual components in the first image, more accurate position information of the visual components can be obtained, thereby saving time for processing the visual components and accelerating the generation of the picture layout.

[0067] In a fifth possible implementation manner of the second aspect, normalizing the position of the visual component includes at least one of the following:

[0068] Adjusting the border of the visual component to a regular quadrilateral;

[0069] Stretching the boundary of the visual component so that the stretched visual component fills the canvas;

[0070] When there is an overlapping area between any two visual components, the overlapping between the any two visual components is eliminated.

[0071] By normalizing the visual component so that its boundary is a regular quadrilateral, filling the entire canvas and without overlapping modules, subsequent visual components can be displayed better on the canvas, thereby obtaining a better target screen layout.

[0072] In a sixth possible implementation manner of the second aspect, the method further includes:

[0073] Perform noise reduction processing on the first image.

[0074] Through the noise reduction process, the interference of the image resolution in the first image is eliminated, so that the recognition process of the visual component in the first image is more accurate.

[0075] In a seventh possible implementation manner of the second aspect, recognizing the first image to obtain the category and position of the visual component includes:

[0076] obtaining a second image based on a valid area of ​​the first image, wherein the second image includes image content within the valid area;

[0077] The second image is identified to obtain the category and position of the visual component.

[0078] When the original image includes an image outside the effective area, the effective area of ​​the original image needs to be identified to obtain the first image, so as to improve the recognition accuracy of the visual components in the image.

[0079] In an eighth possible implementation manner of the second aspect, obtaining the second image based on the valid area of ​​the first image includes any one of the following:

[0080] cropping a valid area of ​​the first image to obtain the second image;

[0081] The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

[0082] The first image is obtained by processing the effective area of ​​the original image, so as to reduce the interference of image content outside the effective area in the original image on subsequent image processing.

[0083] In a third aspect, a data processing device is provided for executing the above data processing method. Specifically, the data processing device includes a functional module for executing the data processing method provided in the first aspect or any optional manner of the first aspect.

[0084] In a fourth aspect, a data processing device is provided for executing the above-mentioned data processing method. Specifically, the data processing device includes a functional module for executing the data processing method provided in the above-mentioned second aspect or any optional manner of the above-mentioned second aspect.

[0085] In a fifth aspect, a computer device is provided, which includes one or more processors and one or more memories, wherein the one or more memories store at least one instruction, which is loaded and executed by the one or more processors to implement the operations performed by the above-mentioned data processing method.

[0086] In a sixth aspect, a computer-readable storage medium is provided, in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement the operations performed by the above-mentioned data processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a specific implementation environment of a data processing method provided by an embodiment of the present disclosure;

[0088] Figure 2 is a block diagram of a computer device 200 according to an exemplary embodiment;

[0089] Figure 3 is a system framework diagram according to an exemplary embodiment;

[0090] Figure 4 is an interactive flow chart showing a data processing method according to an exemplary embodiment;

[0091] Figure 5 is a data schematic diagram showing a data processing method according to an exemplary embodiment;

[0092] Figure 6 is a comparison diagram of an original image and a first image according to an exemplary embodiment;

[0093] Figure 7 The figure shows the positioning result of the target area in the first image after target detection;

[0094] Figure 8 The figure shows the training process of the visual component classification model and the identification process of the visual component category;

[0095] Figure 9 The left side shows an example of a picture visual component, and the right side shows an example of an announcement visual component.

[0096] Figure 10 is a schematic diagram showing module boundary adjustment according to an exemplary embodiment;

[0097] Figure 11 is a schematic diagram showing module boundary stretching according to an exemplary embodiment;

[0098] Figure 12 is a schematic diagram of module coverage according to an exemplary embodiment;

[0099] Figure 13 FIG. 1 is a schematic diagram of the processing process of the first image;

[0100] Figure 14 Shown is a feedback diagram of a visual component;

[0101] Figure 15 is an interactive flow chart illustrating a data processing method according to an exemplary embodiment;

[0102] Figure 16 is a diagram of a data processing device provided by an embodiment of the present disclosure;

[0103] Figure 17 2 is a diagram of a data processing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0104] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0105] Figure 1 This is a specific implementation environment of a data processing method provided by the embodiment of the present disclosure, see Figure 1 The specific implementation environment of the data processing method includes: a server cluster 101 and a computer device 102.

[0106] Server cluster 101, which may include at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server cluster 101 is configured to provide backend services for applications supporting data processing. For example, server cluster 101 may be configured to identify the category of a visual component in a first image during data processing and transmit the identification result to computer device 102, so that computer device 102 can display the result based on the identification result.

[0107] In one possible implementation, the recognition of the visual component category can be performed by a trained recognition model, and accordingly, the server cluster 101 can include a server 1011 for recognition and a server 1012 for model training. Of course, the two servers can also be implemented on the same set of hardware, which is not limited in this embodiment of the present disclosure.

[0108] Computer device 102 is connected to server cluster 101 via a wireless network or a wired network. Computer device 102 can be at least one of a smart phone, a desktop computer, a tablet computer, and a laptop computer. Computer device 102 can serve as an image provider. A user can import an image on computer device 102, then send the image to server cluster 101 for recognition, and then display it based on the recognition result returned by server cluster 101. Of course, the computer device 102 can also independently serve as an image provider and image identifier. That is, a user can import an image on computer device 102, then have computer device 102 recognize the image, and then display it based on the recognition result without having to interact with server cluster 101 in real time. Server cluster 101 can serve as a provider of a model used during recognition. Computer device 102 can download the recognition model provided by server cluster 101 at any time, thereby realizing the above-mentioned image recognition based on the recognition model, and finally displaying it based on the recognition result.

[0109] Computer device 102 may generally refer to one of multiple computer devices. This embodiment uses computer device 102 as an example. Those skilled in the art will appreciate that the number of computer devices may be greater or lesser. This embodiment does not limit the number or type of computer devices.

[0110] Figure 2 2 is a block diagram of a computer device 200 according to an exemplary embodiment. For example, the computer device 200 can be provided as a user-side device or a server. Figure 2 Computer device 200 includes a processing component 201, which further includes one or more processors, and a memory resource represented by memory 202 for storing program code, such as an application, that can be executed by processing component 201. The application stored in memory 202 may include one or more modules, each of which corresponds to a set of program code. In addition, processing component 201 is configured to execute the program code to perform the above-mentioned data processing method.

[0111] The computer device 200 may further include a power supply component 203 configured to perform power management of the computer device 200, a wired or wireless network interface 204 configured to connect the computer device 200 to a network, and an input / output (I / O) interface 205. The computer device 200 may operate based on an operating system stored in the memory 202, such as Windows Server 200. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0112] Next, combine Figure 3 The process shown in the figure briefly introduces the data processing process: first, the complete design prototype of the design department, the high-definition image of the prototype design on the Internet, the template image in the visualization platform, or the photographed image of the large-screen finished product is input as the input image. The scene processing function preprocesses the input image, such as noise processing, effective area identification and cropping, etc., and then the target detection function performs target detection to obtain the regional image, coordinate attributes, and category attributes of the visual component. The image processing function then performs color extraction, layout normalization, and other processing, and the data obtained after the above processing is packaged with metadata. Finally, the computer equipment can perform arrangement based on the metadata obtained by the encapsulation. The arrangement process is through metadata parsing and automatic generation of components, and the development state layout and components can be finally displayed.

[0113] Below, a possible implementation of the data processing method is described based on the process of generating the entire screen layout through interaction between a computer device and a server:

[0114] See also Figure 4 and Figure 5 ,Should Figure 4 is an interactive flow chart showing a data processing method according to an exemplary embodiment. Figure 5 This is a data schematic diagram showing a data processing method according to an exemplary embodiment, which specifically includes the following steps:

[0115] 401. A computer device obtains a first image, where the first image is an image of a screen design drawing.

[0116] The screen design drawing may be a screen design drawing hand-drawn by a user, a prototype image of the screen design drawing, a complete image of the screen design drawing on the Internet, a photographic image of the screen layout, etc., and the embodiments of the present disclosure do not limit this.

[0117] In step 401, the computer device may obtain an original image by photographing or scanning the actual screen design drawing, and save the original image to the computer device. The computer device has an application installed on it that is used to automatically generate the target screen layout. The user imports the original image saved in the computer device into the application, thereby allowing the computer device to obtain the original image.

[0118] In the embodiment of the present disclosure, an electronic image obtained by photographing or scanning may include areas outside the screen design. Therefore, it is necessary to first identify the valid area of ​​the original image and determine the valid area in the original image.

[0119] The above identification of the valid area in the original image can be implemented in any of the following ways:

[0120] Method 1: Identify the valid area in the original image based on Canny edge detection.

[0121] Among them, Canny edge detection refers to extracting edge information from an image. The area composed of edge information is the area with actual image content, that is, the valid area, and the area outside the area composed of edge information is the invalid area.

[0122] The first method specifically includes: performing edge detection on the original image to obtain at least one region, and using a region in the at least one region whose region shape meets the target condition as the effective region.

[0123] The original image is gray-scaled to obtain a first gray-scale image; the first gray-scale image is Gaussian filtered to remove noise in the image; the image after Gaussian filtering is subjected to Canny edge detection; and according to the edge detection result, a quadrilateral area is selected as the area of ​​the paper.

[0124] Among them, the step of "performing Canny edge detection on the Gaussian filtered image" in the above-mentioned method 1 may include the following steps: using a Gaussian filter to process the original image to obtain a smoothed image of the original image. This process filters out the noise of the original image and may amplify the edges of the original image, thereby reducing the recognition of pseudo edges in the original image to a certain extent; calculating the gradient intensity and gradient direction of each pixel in the original image, suppressing the non-maximum values ​​therein, thereby eliminating the stray response caused by edge detection; applying a dual threshold to detect the true edges and pseudo edges of the original image, where the edge composed of pixels with gradient values ​​higher than a first threshold is a true edge, and the edge composed of pixels with gradient values ​​lower than a second threshold is a pseudo edge, wherein the first threshold is greater than the second threshold, and the pixels with gradient values ​​between the first threshold and the second threshold are suppressed. Furthermore, it is also possible to detect whether the pseudo edge is connected to the true edge. When the pseudo edge is connected to the true edge, the pseudo edge is a true edge. When the pseudo edge is not connected to the true edge, the pseudo edge is a false edge. Through the above-mentioned further detection process, the situation where some pseudo edges may also be true edges is avoided, thereby improving the accuracy of edge detection.

[0125] Method 2: Effective area identification based on thresholding.

[0126] The second method specifically includes: performing binarization processing on the original image to obtain at least one white area, and using the white area with a shape that meets the target condition in the at least one white area as the effective area.

[0127] The specific recognition method is: according to the result of grayscale processing of the original image, the grayscale value of each pixel in the original image is obtained, the grayscale values ​​of all pixels are sorted, the median of the grayscale values ​​is obtained, the pixel corresponding to the median of the grayscale value is set to white, and the pixel corresponding to the remaining grayscale values ​​is set to black to obtain at least one white area, the at least one white area is screened, and the white area whose area shape meets the target conditions is regarded as the valid area.

[0128] Wherein, in any of the above-mentioned methods for identifying a valid area, the area shape meeting the target condition means that the shape of the area is a quadrilateral.

[0129] It should be noted that the two aforementioned methods for identifying effective areas can effectively cover the identification of effective areas for the vast majority of screen design drawings. When identifying the effective area of ​​the original image of a screen design drawing, any of the three aforementioned methods can be selected, or they can be used sequentially. If the first method fails to identify the effective area, the next method is triggered to identify the effective area. This is not limited in the present embodiment.

[0130] In the embodiments of the present disclosure, after obtaining the valid area of ​​the original image by the above two methods, the user needs to confirm that the valid area is the correct valid area by operating the computer device. This can be achieved in any of the following ways:

[0131] In one implementation, after the computer device determines the valid area of ​​the original image, it marks the original image. The mark can be a graphic box. The computer device displays the marked original image. When the user determines that the marked valid area is accurate, a confirmation instruction can be triggered through a confirmation operation. After receiving the confirmation instruction, the computer device can determine the identified valid area as the valid area of ​​the original image, so as to execute subsequent steps based on the valid area.

[0132] In one implementation, when the user determines that the marked valid area is inaccurate, the valid area can be determined by manually marking on a computer device. For example, the manual marking method can be: the user marks four points on the original image displayed by the computer device through any input device, and the quadrilateral formed by the lines connecting the four points is the valid area of ​​the original image. The computer device determines the valid area of ​​the original image based on the point coordinates.

[0133] Through the above-mentioned recognition of the effective area, the interference of invalid information in the original image can be eliminated, making subsequent recognition more accurate. Especially for the scenario where the original image is a hand-drawn design drawing, the influence of non-paper areas, such as the background area captured by the photo, can be eliminated, thereby greatly improving the accuracy of subsequent recognition.

[0134] In addition, it should be noted that the effective area identification step is an optional step. During the data processing process, the effective area identification may not be performed, but the image may be sent directly to the server based on the acquired image. Of course, the decision of whether to perform this step may be further made based on the judgment of the image. For example, after the computer device acquires the original image, the source category of the original image may be determined first. For example, if the original image is a photographed image of a hand-drawn design drawing, the effective area identification process is performed; if the original image is a scanned image of a hand-drawn design drawing, the effective area identification step does not need to be performed.

[0135] In the embodiment of the present disclosure, after the computer device identifies the valid area of ​​the original image, the specific method of obtaining the first image based on the valid area can adopt any of the following implementation methods:

[0136] Implementation method 1: cropping the effective area of ​​the original image to obtain a first image.

[0137] After the computer device receives the instruction to confirm cropping or obtains the original image after calibrating four points, it can directly crop the effective area of ​​the original image to obtain an image of the effective area, which is the first image.

[0138] Implementation method 2: cropping the effective area of ​​the original image to obtain an effective area image, and stretching the effective area image to obtain a first image.

[0139] After the computer device receives the instruction to confirm the cropping or obtains the original image after the four points are calibrated, the effective area of ​​the original image is cropped to obtain an image of the effective area. The image of the effective area may not be a regular quadrilateral, so it needs to be stretched to obtain the stretched image of the effective area, which is the first image.

[0140] The stretching processing of the image of the above-mentioned effective area is also the perspective transformation of the image of the effective area. Perspective transformation refers to the use of the condition that the perspective center, image point and target point are collinear, and the image receiving surface (perspective surface) is rotated around the trace line (perspective axis) by a certain angle according to the law of perspective rotation, thereby destroying the original projection light beam and still keeping the projected geometric figure on the image receiving surface unchanged.

[0141] Of course, the step of cropping and stretching the effective area is also an optional step. That is, for the disclosed embodiment, after the above-mentioned computer device obtains the original image, it can identify and detect the effective area of ​​the original image to obtain the first image, or it can directly use the original image as the first image without performing the above steps of identifying and cropping the effective area.

[0142] For example, see Figure 6 As shown, Figure 6 The upper picture is an example of the display of the original image, and the lower picture is the first image obtained after the effective area is identified and processed such as cropping and stretching. The first image includes the image content in the effective area.

[0143] 402. The computer device sends the first image to the server.

[0144] In step 402, after the computer device obtains the first image, it may automatically send the first image to the server, or it may send the first image to the server after detecting the user's confirmation sending instruction, which is not limited in the embodiment of the present disclosure.

[0145] 403. After receiving the first image sent by the computer device, the server identifies the visual component in the first image to obtain the category and position of the visual component in the first image.

[0146] In step 403, after the server receives the first image sent by the computer device, it may perform noise reduction processing on the first image, as the first image may be affected by image resolution and image background color. The noise reduction processing may be Gaussian filtering or median filtering to remove noise interference in the first image, which is not limited in the present embodiment. For example, the process of using median filtering for noise reduction processing is as follows:

[0147] Grayscale processing is performed on the first image to obtain the grayscale value of each pixel in the first image after grayscale processing, and the grayscale values ​​of all pixels in the area where each pixel is located are sorted to obtain the median of the grayscale values ​​of all pixels in the area. The grayscale value represented by the median is the grayscale value of the pixel.

[0148] In an embodiment of the present disclosure, identifying a visual component in a first image to obtain the category and position of the visual component includes the following steps: inputting the first image into a visual component classification model, performing target detection on multiple candidate areas in the first image through the visual component classification model, and obtaining the category and position of the visual component.

[0149] Among them, target detection technology can be used to identify the visual components in the first image to obtain the position and category of the visual components in the first image. The essence of target detection is to perform multi-target positioning on the first image. Multiple candidate areas are selected in the first image by sliding windows or selective search. The candidate areas include at least one visual component. The visual components in the multiple candidate areas are classified and predicted by the visual component classification model, thereby determining the category and position of the visual components in the first image. Figure 7 Shown is the result of locating the target area in the first image after target detection.

[0150] For example, in an embodiment of the present disclosure, the Faster-RCNN target detection model and the inceptionV3 visual component classification model are used to classify the visual components in the first image, specifically including the following steps: obtaining the position and corresponding category of the visual component in the first image through the Faster-RCNN target detection model; filtering the first image through the detected target area to extract the undetected target area and the target area with a detection probability lower than the preset probability; inputting the undetected target area and the target area with a detection probability lower than the preset probability into the inceptionV3 visual component classification model to obtain the category and position of the visual component in the undetected target area and the target area with a detection probability lower than the preset probability; combining the recognition results of the Faster-RCNN target detection model and the inceptionV3 visual component classification model to obtain the category and position of the visual component with higher accuracy.

[0151] The visual component classification model can be trained in the server in advance, such as Figure 8The figure shows the training process of the visual component classification model and the identification process of the visual component category. The user can draw multiple visual components of different categories in advance. The categories of the visual components can be bar charts, line charts, maps, pie charts, liquid level charts, etc., and label the multiple visual components of different categories. For example, the visual components can be labeled using LabelImg (an image labeling tool) to obtain a labeling file. The format of the labeling file can be XML format. Data enhancement is performed on the multiple visual components of different categories. This data enhancement is mainly to reduce the overfitting phenomenon of the network. Overfitting refers to making the hypothesis overly strict in order to obtain a consistent hypothesis. Avoiding overfitting is a core task in classifier design. If a hypothesis can obtain a better fit than other hypotheses on the training data, but cannot fit the data well on a data set outside the training data, then this hypothesis is considered to have overfitted. The data enhancement methods include: rotation, affine transformation, perspective transformation, erosion and expansion, HSV perturbation, gamma perturbation, etc. to simulate changes in lighting and angle. The data-enhanced visual components and pre-drawn visual components are combined into image data, which is annotated. After data enhancement, the dataset is divided into a test set and a training set according to a 1:3 comparison ratio, stored in a file system, and the image data is trained offline to obtain a visual component classification model. The first image is input into the visual component classification model, which divides the first image into regions to obtain the categories of the visual components in each region. The classification results are encapsulated, and the user provides feedback on whether the classification is correct. The feedback results and the image data are combined to form the full image data. The visual component classification model is trained online to obtain a visual component classification model with better generalization ability. The offline training of the visual component classification model is performed on the server. There are many models for object detection, such as region-based convolutional neural networks (RCNN), single shot multibox detector (SSD), and you only look once (YOLO). The first image is input into the visual component classification model, and the first image is recognized by the trained visual component classification model to obtain the category of the visual component in the first image.

[0152] It should be noted that when marking the categories of visual components, the areas with obvious visual component features can be marked, and the visual components with unclear features (such as the picture visual component and the announcement visual component) can be unified as placeholders. A placeholder means that a fixed position is occupied first, and appropriate visual components are added to the position, such as Figure 9The left side shows an example of an image displayable component, and the right side shows an example of an announcement displayable component.

[0153] It should be noted that, see Figure 5 The server may include a recognition server and a GPU server, wherein the GPU server may be used to train and update the visual component classification model, thereby sending the visual component classification model to the recognition server for use.

[0154] In step 403, the position of the visual component may be normalized to obtain a normalized position of the visual component, so as to ensure that the obtained visual component has a regular shape and is suitable for screen display. The step of normalizing the position of the visual component may specifically include any of the following:

[0155] 4031. Adjust the border of the visual component to a regular quadrilateral.

[0156] The adjustment may be to adjust the height and width of the border of the visual component to obtain the border of the visual component having a shape of a regular quadrilateral.

[0157] by Figure 10 For example, the adjustment process Figure 10 The height of the border of the visual component is adjusted. The number of melanin in the border of the visual component is detected from left to right. When the number of melanin on the right is greater than that on the left, the detection continues to the right. When the number of melanin on the right is less than that on the left, the height of the border of the visual component is adjusted. Figure 10 As shown, when the number of melanin pigments on the right side is less than that on the left side for the first time, the heights of the visual component's boundaries A, B, and D are adjusted according to a certain ratio and marked as adjusted after adjustment. When the number of melanin pigments on the right side is less than that on the left side for the second time, since the visual component's boundary B has already been adjusted, detection continues to the right. When the number of melanin pigments on the right side is less than that on the left side for the third time, the visual component's boundary A has already been adjusted. Below the visual component's boundary A, the heights of the visual component's boundaries C and E are adjusted according to a certain ratio. Adjusting the width of the visual component's boundary is similar to adjusting the height of the visual component's boundary and will not be repeated here.

[0158] 4032. Stretch the boundary of the visual component so that the stretched visual component fills the canvas.

[0159] In step 403, the boundary of the visual component is stretched. The specific steps for stretching the boundary of the visual component to fill the canvas are as follows:

[0160] According to the coordinate information of the boundaries of the visual components from small to large, the boundaries of each visual component are stretched in the order of upward, left, downward, and right, until they reach the boundaries of other adjacent visual components or the boundaries of the entire image, such as Figure 11 Shown is a diagram of a displayable's bounds stretching.

[0161] 4033. When there is an overlapping area between any two visual components, eliminate the overlapping between any two visual components.

[0162] In order to ensure that the display is suitable for the screen, the position of the target area in the image can be used to determine whether there is overlap between the target areas. If there is overlap, the position of the target area is adjusted until there is no overlap between the target areas after adjustment. The specific method is as follows:

[0163] When there is an overlapping area between any two target areas and the coverage is relative, that is, the two target areas have overlapping areas on the X axis or Y axis, and the overlapping length is less than 1 / 3 of their respective widths, such as Figure 12 Target areas B and E in the second image overlap on the X-axis. Vertex coordinates 1 and 2 of target area B and vertex coordinates 5 and 6 of target area E are obtained to determine the relative coverage length L, where L = X1 - X3. Target area B is then moved to the left and to the right by a distance equal to L / 2, ensuring that there is no overlap between target areas B and E on the X-axis. A relative coverage check is performed on each target area in the second image to ensure that no two target areas overlap.

[0164] When there is an overlapping area between any two target areas and the coverage is absolute, that is, there are common pixels between the two target areas with absolute coverage, such as Figure 12 Target regions B and C in the second image are obtained. Vertex coordinates 1 and 2 of target region B and vertex coordinates 3 and 4 of target region C are obtained to obtain a coverage length L, where L = X1 - X2. Target region B is shifted to the left by a certain length, and target region C is shifted to the right by a certain length, where the coverage length is L / 2, so that there are no common pixels between target regions B and C. An absolute coverage check is performed on each target region in the second image to ensure that there is no absolute coverage between any two target regions.

[0165] It should be noted that after any coverage elimination operation is completed, the boundary of the target area can also be adjusted until it is adjusted to a regular quadrilateral. The specific operation is the same as the above-mentioned operation of adjusting the boundary of the target area to a regular quadrilateral, and will not be repeated here.

[0166] Another point that needs to be explained is that the above process is described using the processing of the second image as an example. When steps 402 and 403 are not required, the processing of the first image can be performed similarly, which is not described here in detail.

[0167] In the embodiment of the present disclosure, after obtaining the position and category of the visual component, the color of at least one of the visual component and the first image must be identified, that is, the method can also include any one of the following steps: (1) extracting the color of the visual component in the first image; (2) extracting the background color of the first image.

[0168] Taking the extraction of the color of the visual component and the background color of the first image as an example, the extraction of the color of the visual component in the first image and the extraction of the background color of the first image can be implemented in any of the following ways:

[0169] Method 1: Extract the color of the visual component and the background color of the first image based on the HSV model.

[0170] This method specifically includes: based on the HSV values ​​corresponding to the pixel points in the first image, pixel points with HSV values ​​within a preset range are obtained by screening, the HSV values ​​of the screened pixel points are determined as the color of the visual component, and the HSV values ​​of the pixel points in the edge area of ​​the first image are determined as the background color of the first image.

[0171] The steps of the above-mentioned method 1 may include the following steps: converting the first image into a first matrix, wherein the first matrix includes a plurality of triplets, and each triplet in the first matrix is ​​used to represent the HSV value of a pixel in the first image; extracting a second matrix and a third matrix from the first matrix, wherein the second matrix includes the H value in each triplet, and the third matrix includes the V value in each triplet; screening pixels in the second matrix whose H value is greater than a first preset value and whose V value in the third matrix is ​​less than the first preset value, superimposing the second matrix and the third matrix to obtain a single-channel image, counting the pixels in the single-channel image, and taking the HSV value of the pixel with the most occurrences as the background color of the visual component. Screening pixels in the third matrix whose V value is greater than the first preset value, counting the pixels, and taking the HSV value of the pixel with the most occurrences as the foreground color of the visual component. Pixels in the edge area of ​​the first image are obtained by screening, and the average HSV value of the pixels in the edge area is calculated, and the average value is determined as the background color of the first image.

[0172] For example, the first image is represented by the HSV model to obtain a first matrix, and the H matrix (also the second matrix) and the V matrix (also the third matrix) in the first matrix are extracted. The pixels in the H matrix that are greater than 100 (also the first preset value) and the pixels in the V matrix that are less than 100 (also the first preset value) are screened. The H matrix and the V matrix are superimposed to obtain a single-channel image. The pixels in the single-channel image are counted, and the HSV value of the pixel with the most occurrences is used as the background color of the visual component. The pixels in the V matrix that are greater than 100 (also the first preset value) are screened, and the pixels are counted. The HSV value of the pixel with the most occurrences is used as the foreground color of the visual component. The pixels in the edge area of ​​the first image are obtained by screening, and the average HSV value of the pixels in the edge area is calculated, and the average value is determined as the background color of the first image.

[0173] Method 2: Extract the color of the visual component and the background color of the first image based on the RGB model.

[0174] The second method specifically includes: based on the RGB values ​​corresponding to the pixel points in the first image, the pixel points whose RGB values ​​are within a preset range are obtained by screening, the RGB values ​​of the screened pixel points are determined as the color of the visual component, and the RGB values ​​of the pixel points in the edge area of ​​the first image are determined as the background color of the first image.

[0175] The steps of the second method may include the following steps: converting the first image into a fourth matrix, wherein the fourth matrix includes a plurality of triplets, each triplet in the fourth matrix being used to represent the RGB value of a pixel in the first image; extracting a fifth matrix and a sixth matrix from the fourth matrix, wherein the fifth matrix includes pixels whose RGB values ​​are all less than a first preset value, and the sixth matrix includes pixels whose RGB values ​​are all greater than a second preset value; counting the pixels in the fifth matrix, and using the RGB value of the pixel with the most occurrences as the background color of the visual component; counting the pixels in the sixth matrix, and using the RGB value of the pixel with the most occurrences as the foreground color of the visual component. The edge area in the first image is obtained by screening, and the average RGB value of the pixels in the edge area is calculated, and the average value is determined as the background color of the first image.

[0176] For example, the first image is represented using an RGB model to obtain a fourth matrix. A fifth matrix and a sixth matrix are extracted from the fourth matrix. The fifth matrix includes pixels whose RGB values ​​are all less than 100 (i.e., the first preset value), and the sixth matrix includes pixels whose RGB values ​​are all greater than 120 (i.e., the second preset value). The pixels in the fifth matrix are counted, and the RGB value of the pixel with the most occurrences is used as the background color of the visual component. The pixels in the sixth matrix are counted, and the RGB value of the pixel with the most occurrences is used as the foreground color of the visual component. The edge area of ​​the first image is obtained by screening, and the average RGB value of the pixels in the edge area is calculated, and the average value is determined as the background color of the first image.

[0177] Among them, the HSV model is used to identify the color of the visual component. Compared with the recognition of RGB (R is red, G is green, and B is blue), since the pixel points in each color channel in RGB pixel representation fall on [0,255], the foreground colors of the two visual components are often bright tones. It is difficult to determine which channel the bright tones exist in in the RGB channel, resulting in inaccurate color recognition of the visual component.

[0178] It should be noted that when extracting the color of a visual component, you can choose any one of the two visual component color extraction methods mentioned above, or you can use them in sequence. When the current extraction method cannot extract the color of the visual component, the latter extraction method is triggered to extract the color of the visual component. The embodiments of the present disclosure do not limit this.

[0179] like Figure 13 The figure shows a schematic diagram of the first image processing process, including the process of extracting the color of the visual component in the first image, identifying its position, and extracting the background color of the first image. The color of the visual component in the first image is identified using color recognition based on the HSV model and color recognition based on the RGB model to obtain the color of the visual component and the background color of the first image. The first image is normalized to obtain the position information of the visual component. The color and position information of the visual component are encapsulated as metadata for the visual component for subsequent automatic arrangement of the visual component, thereby generating the image layout of the target screen.

[0180] 404. The server sends the category and location of the visual component to the computer device.

[0181] The server can compose screen layout metadata based on the obtained visual component categories and positions. This metadata can include component category attributes and coordinate attributes. This metadata can be in JSON format. For example, taking a pie chart as an example, the metadata and field meanings are as follows:

[0182] res_message: return message

[0183] res_code: return code

[0184] result: result body

[0185] module_num: The number of visual components parsed

[0186] photo_w: image width

[0187] photo_h: image height

[0188] pie_m: visual component category, here is a multi-pie chart

[0189] x: starting x coordinate of the visual component

[0190] y: Starting y coordinate of the visual component

[0191] w: width of the display component

[0192] h: visual component height

[0193] x_percent: The starting x coordinate percentage of the visual component boundary

[0194] y_percent: The starting y coordinate percentage of the visual component boundary

[0195] w_percent: Percentage of the width of the visual component border

[0196] h_percent: Percentage of the visual component's border height

[0197] ind: index value

[0198] score: classification result score

[0199] In the embodiment of the present disclosure, the server may further send the color of the visual component and the background color of the first image to the computer device. For example, taking a pie chart as an example, the meanings of the fields of the color of the visual component and the background color of the first image are as follows:

[0200] m_back_clr: visual component background color

[0201] m_front_clr: visual component foreground color

[0202] back_clr: background color of the first image

[0203] 405. The computer device receives the category and position of the visual component, creates the visual component in the canvas according to the category and position of the visual component, and obtains the screen layout of the target screen.

[0204] In step 405, the computer device transmits the received metadata to the layout engine. The layout engine parses the metadata and arranges the layout of the entire large screen, the position, category and color of the visual components according to the metadata parsing results to obtain the screen layout of the target screen.

[0205] In the embodiment of the present disclosure, the computer device receives the position, category and color of the visual component sent by the server, and provides feedback on whether the result of the classification of the visual component in the first image is correct based on the visual component classification model in the server, and sends the feedback result to the server. The feedback result includes but is not limited to the category of the visual component, the index value of the visual component category prediction, and the accuracy rate of the visual component category prediction, such as Figure 14 The following diagram illustrates visual component feedback. The server updates the visual component classification model based on feedback from the computer device. Upon acquiring an updated visual component classification model, the server distributes the model to the prediction servers to ensure its reliability. The server can encapsulate the visual component feedback sent by the computer device into an incremental visual component library. This library, combined with the base visual component library, forms the full visual component library. This full visual component library is then used by the visual component classification model to identify the categories of visual components in the module.

[0206] It should be noted that the canvas may be created before the computer device sends the first image to the server, or after the computer device receives the module boundary information and the category of the visual component, which is not limited in this embodiment of the present disclosure.

[0207] 406. The computer device displays the screen layout of the target screen.

[0208] In step 406, the target screen layout is displayed on the computer device. Furthermore, the computer device is configured with business data, which can be associated with the business data and displayed in the target screen layout. Business data refers to data owned by the user, such as power consumption data from a power supply bureau or patient data from a hospital.

[0209] In an embodiment of the present disclosure, after the computer device displays the screen layout of the target screen, if the computer device user finds that the screen layout is inconsistent with the expected implementation after viewing the screen layout, the screen layout can be adjusted based on the screen design drawing and the adjusted screen layout can be displayed. If the adjusted screen layout is still inconsistent with the expectation, the screen design drawing can be redesigned.

[0210] The method provided by the embodiment of the present disclosure is that a computer device sends a first image to a server, the server identifies the first image and obtains the category and position of the visual component in the first image, and sends the category and position of the visual component to the computer device. The above process can directly identify the image containing the design draft, thereby obtaining the division of the target area in the image and the category of the visual component in the target area, and automatically generates the screen layout based on the recognition result without the need for manual operation by the user, thereby reducing the waste of time cost. Furthermore, for the design drawing with a complete description of the visual component, the color of the visual component can also be identified, so that the generated screen layout is colorful, which also reduces the waste of time cost.

[0211] See also Figure 15 ,Should Figure 15 This is an interactive flow chart of a data processing method according to an exemplary embodiment, which is different from the above Figure 4 In the embodiment shown, the computer device and the server interact to complete the entire screen layout generation process. Figure 15 The corresponding data processing method is to have the computer device independently complete the image acquisition, recognition and subsequent display processes, thereby completing the screen layout generation process, which specifically includes the following steps:

[0212] 1501. A computer device obtains a first image, where the first image is an image of a screen design drawing.

[0213] If the computer device has image recognition capability, the computer device may directly execute the subsequent step 1502 after acquiring the first image without sending the first image to the server.

[0214] 1502. The computer device identifies the visual component in the first image and obtains the category and position of the visual component in the first image.

[0215] Step 1502 is similar to step 402 above and will not be described again here.

[0216] In step 1502, the computer device identifies the visual component in the first image, obtains the category of the visual component, and then provides feedback on whether the classification result of the visual component is correct based on the visual component classification model, and sends the feedback result to the visual component classification model. The feedback result includes but is not limited to the category of the visual component, the target area corresponding to the visual component, the index value of the visual component category prediction, and the accuracy of the visual component category prediction. The visual component classification model updates the visual component classification model based on the feedback result to ensure the reliability of the visual component classification model.

[0217] 1503. The computer device creates the visual component in the canvas according to the category and position of the visual component to obtain the screen layout of the target screen.

[0218] In step 1503, the server does not need to participate in the generation process of the target screen's picture layout. The computer device can directly create the visual component in the canvas according to the category and position of the visual component, thereby obtaining the picture layout of the target screen.

[0219] 1504. The computer device displays the screen layout of the target screen.

[0220] In the method provided by the embodiment of the present disclosure, a computer device acquires a first image, identifies the visual component in the first image, obtains the category and position of the visual component, and creates the visual component in the canvas according to the category and position of the visual component, thereby obtaining the screen layout of the target screen. The above process can directly identify the image containing the design drawing, thereby obtaining the position and category of the visual component in the image, and generate the screen layout based on the recognition result without the need for manual operation by the user, thereby reducing the waste of time cost. Furthermore, for the design drawing with a complete description of the visual component, the color of the visual component can also be identified, so that the generated screen layout is colorful, which also reduces the waste of time cost.

[0221] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0222] Figure 16 FIG is a diagram of a data processing device provided by an embodiment of the present disclosure. Figure 16 , the device comprises:

[0223] Receiving module 1601, configured to execute step 403 above. The receiving module 1601 is also configured to execute step 405 above.

[0224] Identification module 1602, configured to execute step 403 above;

[0225] The sending module 1603 is used to execute the above step 402. The sending module 1603 is also used to execute the above step 404.

[0226] In one possible implementation, the recognition module 1602 is used to execute the above step 403 of inputting the first image into the visual component classification model, performing target detection on multiple candidate areas in the first image through the visual component classification model, and obtaining the category and position of the visual component.

[0227] In a possible implementation, the receiving module 1601 is further configured to update the visual component classification model according to the feedback result received from the computer device regarding the category of the visual component in the first image in step 404 .

[0228] In a possible implementation, the apparatus further includes an extraction module configured to perform at least one of the following in step 403:

[0229] extracting a color of a visual component in the first image;

[0230] The background color of the first image is extracted.

[0231] In a possible implementation, the device further includes:

[0232] The normalization module is used to perform the normalization of the position of the visual component in step 403 .

[0233] In one possible implementation, the normalization module is configured to perform at least one of the following in step 403:

[0234] Adjust the boundaries of the visual component to a regular quadrilateral;

[0235] Stretch the boundaries of the visual component so that the stretched visual component fills the canvas;

[0236] Removes the overlap between any two displayables if there is an overlap between them.

[0237] In a possible implementation, the device further includes:

[0238] The noise reduction module is used to perform noise reduction processing on the first image in step 403.

[0239] In a possible implementation, the device further includes:

[0240] The acquisition module is used to execute the computer device to acquire the first image in the above step 401; the acquisition module is also used to execute the above step 401 to obtain the second image based on the valid area of ​​the first image.

[0241] The above-mentioned device sends a first image from a computer device to a server. The server identifies the first image and obtains the category and position of the visual component in the first image, and sends the category and position of the visual component to the computer device. The above-mentioned process can directly identify the image containing the design draft, thereby obtaining the division of the target area in the image and the category of the visual component in the target area, and automatically generates the screen layout based on the recognition result without the need for manual operation by the user, thereby reducing the waste of time and cost. Furthermore, for design drawings with complete descriptions of visual components, the color of the visual component can also be identified, so that the generated screen layout is colorful, which also reduces the waste of time and cost.

[0242] Figure 17 FIG is a diagram of a data processing device provided by an embodiment of the present disclosure. Figure 17 , the device comprises:

[0243] Acquisition module 1701, used to execute the above step 1501;

[0244] Identification module 1702, configured to execute step 1502 above;

[0245] Create module 1703 for executing the above step 1503.

[0246] In one possible implementation, the recognition module is used to execute the above step 1502 of inputting the first image into a visual component classification model, performing target detection on multiple candidate areas in the first image through the visual component classification model, and obtaining the category and position of the visual component.

[0247] In a possible implementation, the device further includes:

[0248] The sending module is configured to execute the step 1502 above and, when receiving the feedback result on the category of the visual component in the first image, send the feedback result to the server.

[0249] In a possible implementation, the apparatus further includes an extraction module configured to perform at least one of the following in step 1502:

[0250] extracting a color of a visual component in the first image;

[0251] The background color of the first image is extracted.

[0252] In a possible implementation, the device further includes:

[0253] A normalization module, configured to perform normalization of the position of the visual component in step 1502;

[0254] In one possible implementation, the normalization module is configured to perform at least one of the following in step 1502:

[0255] The border of the visual component is adjusted to a regular quadrilateral;

[0256] Stretch the boundaries of the visual component so that the stretched visual component fills the canvas;

[0257] Removes the overlap between any two displayables if there is an overlap between them.

[0258] In a possible implementation, the device further includes:

[0259] The noise reduction module is used to perform noise reduction processing on the first image in step 1502.

[0260] In a possible implementation, the device further includes:

[0261] The image processing module is used to execute the above step 1501 based on the valid area of ​​the first image to obtain a second image.

[0262] The above-mentioned device obtains a first image by a computer device, identifies the visual components in the first image, obtains the category and position of the visual components, creates the visual components in the canvas according to the category and position of the visual components, and thus obtains the screen layout of the target screen. The above-mentioned process can directly identify the image containing the design drawing, thereby obtaining the position and category of the visual components in the image, and generates the screen layout based on the recognition results, without the need for manual operation by the user, thereby reducing the waste of time costs. Furthermore, for the design drawing with a complete description of the visual components, the color of the visual components can also be identified, so that the generated screen layout is colorful, which also reduces the waste of time costs.

[0263] It should be noted that the data processing apparatus provided in the above embodiments uses the division of the above functional modules as an example for data processing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus provided in the above embodiments and the data processing method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0264] In an exemplary embodiment, a computer-readable storage medium is further provided, such as a memory including instructions, wherein the instructions can be executed by a processor to perform the data processing method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0265] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0266] The above are merely optional embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a first image, where the first image is an image of a screen design drawing; Identifying a visual component in the first image to obtain a category and a position of the visual component in the first image, where the category of the visual component is a bar chart, a line chart, a map, a pie chart, or a liquid level chart; Normalizing the position of the visual component by at least one of the following: adjusting the boundary of the visual component to a regular quadrilateral, stretching the boundary of the visual component so that the stretched visual component fills the canvas, and eliminating the overlap between any two visual components if there is an overlap between them. The visual components are created in the canvas according to the categories and normalized positions of the visual components to obtain a screen layout of the target screen.

2. The method according to claim 1, characterized in that The identifying the visual component in the first image to obtain the category and position of the visual component in the first image includes: The first image is input into a visual component classification model, and target detection is performed on multiple candidate areas in the first image through the visual component classification model to obtain the category and position of the visual component.

3. The method according to claim 2, characterized in that After performing target detection on multiple candidate areas in the first image using the visual component classification model to obtain the category and position of the visual component, the method further includes: When a feedback result on the category of the visual component in the first image is received, the feedback result is sent to the server.

4. The method according to claim 1, wherein The method further comprises at least one of the following: extracting the color of the visual component in the first image; extracting the background color of the first image; The step of creating the visual component in the canvas according to the category and position of the visual component to obtain the layout of the target screen includes: The visual component is created in a canvas according to the extracted color and the category and position of the visual component to obtain a screen layout of the target screen.

5. The method according to claim 1, characterized in that The method further comprises: Perform noise reduction processing on the first image.

6. The method according to claim 1, characterized in that The step of identifying the first image and obtaining the category and position of the visual component includes: obtaining a second image based on a valid area of ​​the first image, wherein the second image includes image content within the valid area; The second image is identified to obtain the category and position of the visual component.

7. The method according to claim 6, characterized in that The obtaining of the second image based on the valid area of ​​the first image includes any one of the following: cropping a valid area of ​​the first image to obtain the second image; The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

8. A data processing method, characterized in that: The method comprises: receiving a first image sent by a computer device, wherein the first image is an image of a screen design; Identifying a visual component in the first image to obtain a category and a position of the visual component in the first image, where the category of the visual component is a bar chart, a line chart, a map, a pie chart, or a liquid level chart; Normalizing the position of the visual component by at least one of the following: adjusting the boundary of the visual component to a regular quadrilateral, stretching the boundary of the visual component so that the stretched visual component fills the canvas, and eliminating the overlap between any two visual components if there is an overlap between them. The category and the normalized position of the visual component are sent to the computer device, and the category and the normalized position of the visual component are used by the computer device to create the visual component in the canvas to obtain the screen layout of the target screen.

9. The method according to claim 8, characterized in that The identifying the visual component in the first image to obtain the category and position of the visual component in the first image includes: The first image is input into a visual component classification model, and target detection is performed on multiple candidate areas in the first image through the visual component classification model to obtain the category and position of the visual component.

10. The method according to claim 9, characterized in that After performing target detection on multiple candidate areas in the first image using the visual component classification model to obtain the category and position of the visual component, the method further includes: When receiving a feedback result from the computer device on the category of the visual component in the first image, the visual component classification model is updated according to the feedback result.

11. The method according to claim 8, characterized in that The method further comprises at least one of the following: extracting the color of the visual component in the first image; extracting the background color of the first image; The sending the category and position of the visual component to the computer device includes: The extracted color and the category and position of the visual component are sent to the computer device.

12. The method according to claim 8, characterized in that The method further comprises: Perform noise reduction processing on the first image.

13. The method according to claim 8, characterized in that The step of identifying the first image and obtaining the category and position of the visual component includes: obtaining a second image based on a valid area of ​​the first image, wherein the second image includes image content within the valid area; The second image is identified to obtain the category and position of the visual component.

14. The method according to claim 13, wherein: The obtaining of the second image based on the valid area of ​​the first image includes any one of the following: cropping a valid area of ​​the first image to obtain the second image; The effective area of ​​the first image is cropped to obtain an effective area image, and the effective area image is stretched to obtain a second image.

15. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a first image; an identification module, configured to identify a visual component in the first image and obtain a category and position of the visual component in the first image, wherein the category of the visual component is a bar chart, a line chart, a map, a pie chart, or a liquid level chart; a normalization module, configured to normalize the position of the visual component by at least one of the following: adjusting the boundary of the visual component to a regular quadrilateral, stretching the boundary of the visual component so that the stretched visual component fills the canvas, and eliminating the overlap between any two visual components when there is an overlap between the two visual components; The creation module is used to create the visual component in the canvas according to the category and the normalized position of the visual component to obtain the screen layout of the target screen.

16. The device according to claim 15, characterized in that The recognition module is used to input the first image into a visual component classification model, perform target detection on multiple candidate areas in the first image through the visual component classification model, and obtain the category and position of the visual component.

17. The device according to claim 15, characterized in that The device further comprises: The sending module is configured to send the feedback result on the category of the visual component in the first image to the server when the feedback result is received.

18. The device according to claim 15, characterized in that The apparatus further includes an extraction module configured to perform at least one of the following: extracting the color of the visual component in the first image; extracting the background color of the first image; The creation module is used to create the visual component in the canvas according to the extracted color and the category and position of the visual component to obtain the screen layout of the target screen.

19. The device according to claim 15, characterized in that The device further comprises: A noise reduction module is used to perform noise reduction processing on the first image.

20. The device according to claim 15, characterized in that The apparatus further includes an image processing module configured to perform the following steps: obtaining a second image based on a valid area of ​​the first image, wherein the second image includes image content within the valid area.

21. A data processing device, characterized in that: The device comprises: A receiving module, configured to receive a first image sent by a computer device; an identification module, configured to identify the first image and obtain a category and position of a visual component, wherein the category of the visual component is a bar chart, a line chart, a map, a pie chart, or a liquid level chart; a normalization module, configured to normalize the position of the visual component by at least one of the following: adjusting the boundary of the visual component to a regular quadrilateral, stretching the boundary of the visual component so that the stretched visual component fills the canvas, and eliminating the overlap between any two visual components when there is an overlap between the two visual components; A sending module is used to send the category and normalized position of the visual component to the computer device, and the category and normalized position of the visual component are used by the computer device to create the visual component in the canvas to obtain the screen layout of the target screen.

22. The device according to claim 21, characterized in that The recognition module is used to input the first image into a visual component classification model, perform target detection on multiple candidate areas in the first image through the visual component classification model, and obtain the category and position of the visual component.

23. The device according to claim 21, characterized in that The receiving module is further configured to update the visual component classification model according to the feedback result received from the computer device regarding the category of the visual component in the first image.

24. The device according to claim 21, characterized in that The apparatus further includes an extraction module configured to perform at least one of the following: extracting the color of the visual component in the first image; extracting the background color of the first image; The sending module is used to send the extracted color and the category and position of the visual component to the computer device.

25. The device according to claim 21, characterized in that The device further comprises: A noise reduction module is used to perform noise reduction processing on the first image.

26. The device according to claim 21, characterized in that The device further comprises: An acquisition module is used to obtain a second image based on a valid area of ​​the first image.

27. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one instruction is stored in the one or more memories, and the instruction is loaded and executed by the one or more processors to implement the operation performed by the data processing method of any one of claims 1 to 7 or claims 8 to 14.

28. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the operation performed by the data processing method according to any one of claims 1 to 7 or claims 8 to 14.

Images