Element capturing method and device, electronic equipment and storage medium

Abstract

According to the element capturing method and device, the electronic equipment and the storage medium provided by the invention, the obtained to-be-processed image data is input into the trained image segmentation model for image segmentation processing, and multiple pieces of first image data are obtained; performing text extraction processing on the plurality of first image data to obtain target text data; performing matching processing on the target text data from a preset keyword database to obtain a matching result; and under the condition of determining that the target text data is matched with the keyword according to the matching result, determining the first image data corresponding to the target text data as a target element, and performing object attribute packaging processing on the target element to obtain a packaged target element. According to the embodiment of the invention, the to-be-grabbed element is positioned in a manner of carrying out image instance segmentation on the to-be-processed image data, carrying out character extraction on the segmented legend and carrying out secondary matching according to the extracted character, so that related pixels can be reduced, and the recognition efficiency and accuracy during element grabbing are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for element capture, an electronic device, and a storage medium. Background Art

[0002] UiPath is a relatively mature interface element capture technology on the market. Its principle is to parse the HTML document of the web page (World Wide Web, WEB) interface and locate the elements of the WEB interface based on a selector-like language. UiPath can efficiently locate and identify elements when capturing elements of the WEB interface, but the success rate is low for non-WEB interfaces, such as: capturing elements of the QT (Qt Toolkit) interface or the Abstract Window Toolkit (Abstract Window Toolkit, AWT) development interface. However, in reality, there is a large demand for capturing elements from the QT interface or AWT development interface.

[0003] In order to meet the demand of grabbing elements from the QT interface or AWT development interface, the UiPath solution is usually expanded before grabbing elements from the QT interface or AWT development interface. Currently, most of the expansion solutions for UiPath are to grab elements through the image pixel comparison function. For example, if you need to grab the "Query" button, you need to take a screenshot of the query button in the QT interface or AWT development interface in advance, and perform a global pixel comparison between the screenshot and the original image of the QT interface or AWT development interface to determine whether the screenshot is the element that needs to be grabbed.

[0004] However, the method in the prior art of determining whether a screenshot is an element to be captured by performing global pixel comparison between the screenshot and the original image involves too many pixels, has low recognition efficiency, and has low recognition accuracy when encountering a background with an anti-leakage watermark. Summary of the invention

[0005] The present disclosure provides a method and device for element capture, an electronic device and a storage medium. The main purpose is to solve the problem that the method of judging whether a screenshot is an element to be captured by comparing the screenshot with the original image by global pixel comparison in the prior art involves too many pixels, the recognition efficiency is low, and the recognition accuracy is low when encountering a background with an anti-leakage watermark.

[0006] According to a first aspect of the present disclosure, a method for element grabbing is provided, comprising:

[0007] Inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data; wherein the image data to be processed includes predetermined elements to be captured;

[0008] Performing text extraction processing on the plurality of first image data to obtain target text data corresponding to each of the plurality of first image data;

[0009] The target text data is matched from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the element to be captured;

[0010] When it is determined according to the matching result that the target text data matches the keyword, the first image data corresponding to the target text data is determined as a target element, and object attribute encapsulation processing is performed on the target element to obtain an encapsulated target element.

[0011] Optionally, the step of inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data includes:

[0012] Based on the trained image segmentation model, data conversion processing is performed on the image data to be processed to obtain a first pixel vector corresponding to the image data to be processed; wherein the trained image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers;

[0013] Performing vector operation processing on the first pixel vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second pixel vectors, and performing vector splicing processing on the multiple second pixel vectors to obtain a third pixel vector;

[0014] Performing vector convolution processing on the third pixel vector, and performing data activation processing on the third pixel vector after the convolution processing by using a preset activation function to obtain a fourth pixel vector;

[0015] The fourth pixel vector is subjected to bilinear interpolation processing to obtain a target pixel vector, and the target pixel vector is subjected to data conversion processing to obtain the plurality of first image data.

[0016] Optionally, performing vector operation processing on the first pixel vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second pixel vectors includes:

[0017] Performing a first convolution process on the first pixel vector to obtain a first convolution vector, and performing a first pooling process on the first convolution vector to obtain a first processing vector;

[0018] Performing a second convolution process on the first convolution vector to obtain a second convolution vector, and performing a second pooling process on the second convolution vector to obtain a second processed vector;

[0019] Performing a third convolution process on the second convolution vector to obtain a third convolution vector, and performing a fourth convolution process on the third convolution vector to obtain a third processed vector;

[0020] The third convolution vector is subjected to a fifth convolution process to obtain a fourth convolution vector, and the fourth convolution vector is subjected to a bilinear upsampling process to obtain a fourth processing vector; wherein the plurality of second pixel vectors include at least a first processing vector, a second processing vector, a third processing vector, and a fourth processing vector.

[0021] Optionally, before inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data, the method further includes:

[0022] Determine an application scenario corresponding to the trained image segmentation model, and obtain a preset amount of training image data based on the application scenario; wherein different application scenarios correspond to different training image data;

[0023] Performing image annotation processing on the training image data to obtain training sample data, and performing data enhancement processing on the training sample data to obtain target training data;

[0024] The target training data is input into the image segmentation model for model training processing to obtain the trained image segmentation model, wherein the image segmentation model is a pre-set deep learning model.

[0025] Optionally, the step of inputting the target training data into an image segmentation model for model training to obtain the trained image segmentation model comprises:

[0026] Based on the image segmentation model, the target training data is subjected to data conversion processing to obtain a first training vector and a label vector corresponding to the target training data; wherein the image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers;

[0027] Performing vector operation processing on the first training vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second training vectors, and performing vector concatenation processing on the multiple second training vectors to obtain a third training vector;

[0028] Performing vector convolution processing on the third training vector, and performing data activation processing on the third training vector after the convolution processing by using a preset activation function to obtain a fourth training vector;

[0029] Performing bilinear interpolation processing on the fourth training vector to obtain a target training vector, and performing data calculation processing on the target training vector and the label vector using a preset loss function to obtain a target loss value;

[0030] Repeat the above steps until the target loss value is less than the preset loss threshold, and then obtain the trained image segmentation model.

[0031] Optionally, after matching the target text data from a preset keyword database to obtain a matching result, the method further includes:

[0032] When it is determined according to the matching result that the target text data does not match the keyword, it is determined that the first image data corresponding to the target text data is an error element.

[0033] According to a second aspect of the present disclosure, a device for element grabbing is provided, comprising:

[0034] A segmentation unit, used for inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data; wherein the image data to be processed includes predetermined elements to be captured;

[0035] An extraction unit, configured to perform text extraction processing on the plurality of first image data to obtain target text data corresponding to each of the plurality of first image data;

[0036] A matching unit, used for matching the target text data from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the elements to be captured;

[0037] a determining unit, configured to determine, when it is determined according to the matching result that the target text data matches the keyword, that the first image data corresponding to the target text data is a target element;

[0038] The encapsulation unit is used to perform object attribute encapsulation processing on the target element to obtain the encapsulated target element.

[0039] Optionally, the segmentation unit includes:

[0040] A conversion module, used to perform data conversion processing on the image data to be processed based on the trained image segmentation model to obtain a first pixel vector corresponding to the image data to be processed; wherein the trained image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers;

[0041] A processing module, configured to perform vector operation processing on the first pixel vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second pixel vectors;

[0042] The processing module is further used to perform vector splicing processing on the plurality of second pixel vectors to obtain a third pixel vector;

[0043] The processing module is further used to perform vector convolution processing on the third pixel vector;

[0044] an activation module, used for performing data activation processing on the third pixel vector after the convolution processing by using a preset activation function to obtain a fourth pixel vector;

[0045] The processing module is further used to perform bilinear interpolation processing on the fourth pixel vector to obtain a target pixel vector, and perform data conversion processing on the target pixel vector to obtain the plurality of first image data.

[0046] Optionally, the processing module is further used for:

[0047] Performing a first convolution process on the first pixel vector to obtain a first convolution vector, and performing a first pooling process on the first convolution vector to obtain a first processing vector;

[0048] Performing a second convolution process on the first convolution vector to obtain a second convolution vector, and performing a second pooling process on the second convolution vector to obtain a second processed vector;

[0049] Performing a third convolution process on the second convolution vector to obtain a third convolution vector, and performing a fourth convolution process on the third convolution vector to obtain a third processed vector;

[0050] The third convolution vector is subjected to a fifth convolution process to obtain a fourth convolution vector, and the fourth convolution vector is subjected to a bilinear upsampling process to obtain a fourth processing vector; wherein the plurality of second pixel vectors include at least a first processing vector, a second processing vector, a third processing vector, and a fourth processing vector.

[0051] Optionally, the device further comprises:

[0052] An acquisition unit, used to determine an application scenario corresponding to the trained image segmentation model, and acquire a preset amount of training image data based on the application scenario; wherein different application scenarios correspond to different training image data;

[0053] A labeling unit, used for performing image labeling processing on the training image data to obtain training sample data;

[0054] An enhancement unit, used to perform data enhancement processing on the training sample data to obtain target training data;

[0055] A training unit is used to input the target training data into the image segmentation model for model training processing to obtain the trained image segmentation model, wherein the image segmentation model is a pre-set deep learning model.

[0056] Optionally, the training unit includes:

[0057] A conversion module, configured to perform data conversion processing on the target training data based on the image segmentation model to obtain a first training vector and a label vector corresponding to the target training data; wherein the image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers;

[0058] a processing module, configured to perform vector operation processing on the first training vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second training vectors, and perform vector concatenation processing on the multiple second training vectors to obtain a third training vector;

[0059] The processing module is further used to perform vector convolution processing on the third training vector, and perform data activation processing on the third training vector after the convolution processing through a preset activation function to obtain a fourth training vector;

[0060] The processing module is further used to perform bilinear interpolation processing on the fourth training vector to obtain a target training vector;

[0061] A calculation module, used to perform data calculation processing on the target training vector and the label vector through a preset loss function to obtain a target loss value;

[0062] The repetition module is used to repeat the above steps until the target loss value is less than a preset loss threshold, thereby obtaining the trained image segmentation model.

[0063] Optionally, the determination unit is further configured to, when it is determined according to the matching result that the target text data does not match a keyword, determine that the first image data corresponding to the target text data is an error element.

[0064] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0065] at least one processor; and

[0066] a memory communicatively connected to the at least one processor; wherein,

[0067] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0068] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0069] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0070] The method and device, electronic device and storage medium for element capture provided by the present disclosure input the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain multiple first image data; wherein the image data to be processed includes predetermined elements to be captured; text extraction processing is performed on the multiple first image data to obtain target text data corresponding to each of the multiple first image data; the target text data is matched from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the elements to be captured; when it is determined according to the matching result that the target text data matches the keyword, the first image data corresponding to the target text data is determined to be the target element, and the target element is encapsulated with object attributes to obtain the encapsulated target element. Compared with the related art, the embodiment of the present disclosure performs image instance segmentation on the image data to be processed through a trained image segmentation model, extracts text from the segmented legend, and locates the predetermined elements to be captured by performing secondary matching based on the extracted text, which can reduce the pixels involved. At the same time, the elements to be captured are determined by combining image instance segmentation and text extraction, which can improve the recognition efficiency and accuracy when capturing elements.

[0071] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0073] Figure 1 A flowchart of a method for capturing an element provided by an embodiment of the present disclosure;

[0074] Figure 2 A schematic diagram of a flow chart of image segmentation processing provided by an embodiment of the present disclosure;

[0075] Figure 3 A schematic diagram of the principle of processing data by a model provided in an embodiment of the present disclosure;

[0076] Figure 4 A schematic diagram of a model training process provided by an embodiment of the present disclosure;

[0077] Figure 5 An overall processing logic diagram of an element grabbing method provided by an embodiment of the present disclosure;

[0078] Figure 6 A schematic diagram of the structure of an element grabbing device provided by an embodiment of the present disclosure;

[0079] Figure 7 A schematic diagram of the structure of another element grabbing device provided by an embodiment of the present disclosure;

[0080] Figure 8 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0081] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0082] The following describes the method and apparatus for element capture, electronic device, and storage medium according to embodiments of the present disclosure with reference to the accompanying drawings.

[0083] Figure 1 A flowchart of a method for capturing an element provided in an embodiment of the present disclosure.

[0084] like Figure 1 As shown, the method comprises the following steps:

[0085] Step 101, input the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data; wherein the image data to be processed includes predetermined elements to be captured.

[0086] In the embodiments of the present disclosure, the image data to be processed is a screenshot of a web page interface, including but not limited to: a WEB interface screenshot, a QT interface screenshot, an AWT interface screenshot, etc. The trained image segmentation model is a pre-trained model, and the type of the trained image segmentation model includes but is not limited to: a deep learning neural network, etc. Specifically, the embodiments of the present disclosure do not impose any restrictions on the content of the image data to be processed and the type of the trained image segmentation model.

[0087] Among them, the image segmentation of the image data to be processed requires segmenting each element in the image data to be processed, such as: input box, title, table, header, button and other elements, and each element can have multiple different types, such as: buttons include query, confirm, cancel, etc. When performing image segmentation, the segmentation is performed according to the element edges of each element in the image data to be processed, that is, the first image data is the image data corresponding to each element in the image data to be processed.

[0088] It should be noted that the embodiment of the present disclosure mainly adopts a method combining image instance segmentation and OCR text recognition to determine the specified element, that is, the predetermined element to be grasped, the element to be grasped is the pre-specified element that needs to be grasped, and the description of the relationship between the element to be grasped and the target element, for example: if it is predetermined that the element grasping task this time is: it is necessary to grasp the query button element, then the element to be grasped is the query button element, and then, the target element finally grasped is also the query button element, that is, the element to be grasped is the theoretically specified element expected to be grasped, and the target element is the expected element actually grasped.

[0089] The trained image segmentation model contains image models corresponding to multiple elements, such as image models corresponding to button elements, image models corresponding to input box elements, etc. When performing image segmentation, the corresponding image model in the trained image segmentation model can be called according to the predetermined elements to be captured (elements to be captured), and image segmentation can be performed on the image data to be processed. At this time, the first image data segmented is the image data corresponding to the elements to be captured. Among them, the elements to be captured represent a certain type of elements to be captured (i.e., including elements and types, such as query buttons, etc.). When calling the corresponding image model in the trained image segmentation model, only the image model corresponding to the element is determined, such as the image model corresponding to the button element. The first image data segmented is the image corresponding to the entire button element. Therefore, keyword matching is required later to determine the elements to be captured in the first image data, i.e., the target elements.

[0090] Specifically, when performing image segmentation processing using a trained image segmentation model, the following methods may also be used but are not limited to: determining a first segmentation model corresponding to the element to be captured in the trained image segmentation model based on the element to be captured; wherein the trained image segmentation model includes at least the first segmentation model; performing image segmentation processing on the image data to be processed using the first segmentation model to obtain the multiple first image data corresponding to the element to be captured.

[0091] Step 102: Perform text extraction processing on the plurality of first image data to obtain target text data corresponding to each of the plurality of first image data.

[0092] In the embodiment of the present disclosure, when text is extracted from the first image data, the text extraction tool is a custom-set tool, such as the TesseraOCR tool, etc. TesseraOCR is a Java-based cross-platform optical character recognition (OCR) tool that aims to provide efficient and accurate image text detection and recognition functions.

[0093] The target text data is text extracted from the first image data. For example, when the first image data is a button element, the target text data is text extracted from the button element. When the button element type is a query button, the target text data is query. When the button element type is a confirm button, the target text data is confirm.

[0094] Step 103, matching the target text data from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the elements to be captured.

[0095] In the disclosed embodiment, the preset keyword database is a custom-set database, and there are multiple preset keyword databases. The preset keyword database is a database corresponding to the keywords in each element. For example, a button element corresponds to multiple preset keyword databases. At the same time, the corresponding preset keyword databases are different according to the type of button element, that is, the query button corresponds to a preset keyword database with the keyword being query, the confirm button corresponds to a preset keyword database with the keyword being confirm, and so on.

[0096] It should be noted that when the target text data is matched from a preset keyword database, it is necessary to determine the preset keyword database based on the element to be captured. For example, when the element to be captured is a query button, a match is performed in the preset keyword database where the keyword is query to obtain a matching result.

[0097] Specifically, when the target text data is matched from a preset keyword database, the following method may also be used but is not limited to: determining a corresponding first keyword database based on the element to be captured; wherein the preset keyword database includes at least the first keyword database; matching the target text data from the first keyword database to obtain the matching result.

[0098] Step 104, when it is determined according to the matching result that the target text data matches the keyword, the first image data corresponding to the target text data is determined as a target element, and object attribute encapsulation processing is performed on the target element to obtain an encapsulated target element.

[0099] In an embodiment of the present disclosure, if target text data extracted from a first image data can be matched with a keyword from a preset keyword database, then the first image data is determined to be a target element. For example, the element to be captured is a query button. At this time, the target text data is matched in a preset keyword database corresponding to the query button and whose keyword is the query. If the keyword is matched, then the first image data corresponding to the target text data is determined to be the element to be captured, i.e., the target element.

[0100] Among them, in order to apply the captured target elements, it is necessary to encapsulate the target elements with object attributes. The method of encapsulating the object attributes includes but is not limited to: encapsulating the object attributes according to the WEBCTRL specification. WEBCTRL: The WEBCTRL tag is a web page element selector and is the main tag of UiPath when selecting web page elements. The HTML elements involved in the web page can be quickly captured through the unique tags. The encapsulated target element will add some information, such as coordinates, text, etc.

[0101] The method for element capture provided by the present disclosure includes: inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data; wherein the image data to be processed includes predetermined elements to be captured; performing text extraction processing on the plurality of first image data to obtain target text data corresponding to each of the plurality of first image data; performing matching processing on the target text data from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the elements to be captured; in the case where it is determined according to the matching result that the target text data matches the keyword, determining the first image data corresponding to the target text data as the target element, and performing object attribute encapsulation processing on the target element to obtain the encapsulated target element. Compared with the related art, the embodiment of the present disclosure performs image instance segmentation on the image data to be processed through a trained image segmentation model, extracts text from the segmented legend, and locates the predetermined elements to be captured by performing secondary matching according to the extracted text, which can reduce the pixels involved, and at the same time, the elements to be captured are determined by combining image instance segmentation and text extraction, which can improve the recognition efficiency and accuracy of element capture.

[0102] In one possible implementation of the embodiment of the present disclosure, when performing image segmentation processing on the image data to be processed by using the trained image segmentation model, the image data to be processed is first converted into a vector form, and then the vectors converted from the image data to be processed are processed by various structural layers in the model, and finally the first image data is obtained. Specifically, regarding the image segmentation processing of the image data to be processed, the embodiment of the present disclosure provides a flow chart of the image segmentation processing, such as Figure 2 As shown, including:

[0103] Step 201, based on the trained image segmentation model, perform data conversion processing on the image data to be processed to obtain a first pixel vector corresponding to the image data to be processed; wherein the trained image segmentation model includes at least multiple convolutional layers and multiple pooling layers.

[0104] In the disclosed embodiments, it is necessary to convert the image data to be processed into vector data before further computational processing. The methods for converting the image to be processed into vector data (i.e., the first pixel vector) include but are not limited to: Speeded Up Robust Features (SURF): Similar to SIFT, SURF also extracts key points and generates descriptors, but it calculates faster and can also combine descriptors into vectors. Histogram of Oriented Gradients (HOG): The HOG descriptor describes the shape and texture of an image by calculating the gradient direction distribution of a local area in the image, and finally connects the histograms of all areas into vectors. Convolutional Neural Network (CNN) Feature Extraction: Use pre-trained CNN models, such as VGG, ResNet, etc., to extract image features. Feature vectors, etc. can be obtained by taking the output of a layer in the network (usually the last convolutional layer before the fully connected layer).

[0105] The trained image segmentation model is built based on convolutional layers and pooling layers, and includes multiple convolutional layers and pooling layers.

[0106] Step 202: Perform vector operation processing on the first pixel vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second pixel vectors, and perform vector splicing processing on the multiple second pixel vectors to obtain a third pixel vector.

[0107] In the embodiment of the present disclosure, the number of channels and the stride (strding) of the multiple convolutional layers are custom settings, and the number of channels and the stride (strding) of different convolutional layers may be the same or different, and the stride (strding) of the multiple pooling layers is also custom settings, and the stride (strding) of different pooling layers may be the same or different.

[0108] Regarding the setting of the number of channels and strding of the convolution layer, for example, the number of channels of a certain convolution layer is 16, the strding is 1, the number of channels of a certain convolution layer is 32, the strding is 1, etc. Regarding the setting of the strding of the pooling layer, for example, 4, 2, etc. Specifically, regarding the setting of the number of channels and strding, the embodiment of the present disclosure does not limit it.

[0109] Among them, there are multiple second pixel vectors, and each second pixel vector undergoes different convolution processing and pooling processing, for example: a second pixel vector undergoes one convolution processing and one pooling processing, a second pixel vector undergoes two convolution processing and one pooling processing, a second pixel vector undergoes four convolution processing and no pooling processing, a second pixel vector undergoes four convolution processing and bilinear upsampling processing and no pooling processing, etc.

[0110] When the second pixel vectors are spliced, all the second pixel vectors are spliced ​​to finally obtain a third pixel vector.

[0111] Step 203, performing vector convolution processing on the third pixel vector, and performing data activation processing on the third pixel vector after the convolution processing by a preset activation function to obtain a fourth pixel vector.

[0112] In the embodiment of the present disclosure, the preset activation function is a custom-set function that can improve the expressiveness of the model and introduce sparsity, thereby reducing the model complexity and computational complexity. The setting of the preset activation function, for example, is: ReLU (Rectified Linear Unit), etc.

[0113] Step 204: perform bilinear interpolation processing on the fourth pixel vector to obtain a target pixel vector, and perform data conversion processing on the target pixel vector to obtain the plurality of first image data.

[0114] In the disclosed embodiment, the bilinear interpolation process can be implemented by a built-in function or layer, which is an interpolation method used to estimate an unknown value between two known points. In image processing, the value of a new pixel is calculated by considering the values ​​of the four nearest neighboring pixels around the pixel point, which can be used to initialize the weights of the convolutional layer.

[0115] Among them, regarding the method of converting the target vector into the first image data, for example: vector reshaping: the output vector (target vector) directly corresponds to the pixel value of the image, and the one-dimensional vector can be converted back to the original two-dimensional (grayscale image) or three-dimensional (color image) shape through the reshaping operation.

[0116] In one implementable manner of the embodiment of the present disclosure, when vector operation processing is performed on a first pixel vector through multiple convolution layers and multiple pooling layers, collaborative processing of multiple convolution layers and multiple pooling layers is required. Therefore, in order to accurately and efficiently process the first pixel vector, it can also be implemented in but not limited to the following manner: performing a first convolution processing on the first pixel vector to obtain a first convolution vector, and performing a first pooling processing on the first convolution vector to obtain a first processing vector; performing a second convolution processing on the first convolution vector to obtain a second convolution vector, and performing a second pooling processing on the second convolution vector to obtain a second processing vector; performing a third convolution processing on the second convolution vector to obtain a third convolution vector, and performing a fourth convolution processing on the third convolution vector to obtain a third processing vector; performing a fifth convolution processing on the third convolution vector to obtain a fourth convolution vector, and performing a bilinear upsampling processing on the fourth convolution vector to obtain a fourth processing vector; wherein the multiple second pixel vectors include at least the first processing vector, the second processing vector, the third processing vector, and the fourth processing vector.

[0117] In order to facilitate understanding of the processing process of the image data to be processed, the embodiment of the present disclosure provides a schematic diagram of the principle of model processing data, such as Figure 3 As shown in the figure, the number at the front of the frame represents the number of channels, for example: "16 3*3Conv(1)" represents a convolution layer with 16 channels and a convolution kernel of 3*3; the numbers in brackets represent the strding of the convolution kernel and the pooling layer, for example: Conv(1) represents that the strding of the convolution layer is 1, and Maxpooling(4) represents that the strding of the pooling layer is 4; "3 original image pixel vector" is the first pixel vector and the first pixel vector has three channels; "3 output vector" is the target pixel vector and the target pixel vector has three channels; Conv represents a convolution layer, Maxpooling represents a pooling layer, bilinear upsample represents bilinear upsampling, ReLu represents an activation function, and bilinear interpolation represents bilinear interpolation.

[0118] In one possible implementation of the embodiment of the present disclosure, it is necessary to train the image segmentation model before performing image segmentation, so that the image data to be processed can be processed by the trained image segmentation model. Therefore, in order to accurately segment the image and ensure the accuracy of image segmentation, it is necessary to train the model. Specifically, regarding the training of the model, the embodiment of the present disclosure provides a flow chart of model training, such as Figure 4 As shown, including:

[0119] Step 401, determining the application scenario corresponding to the trained image segmentation model, and acquiring a preset amount of training image data based on the application scenario; wherein different application scenarios correspond to different training image data.

[0120] In the present disclosed embodiment, since it is necessary to ensure that the trained image segmentation model can not only capture web page elements, but also capture elements of some QT or AWT interfaces, and at the same time improve the applicability of the trained image segmentation model, it is necessary to obtain different training image data according to different application scenarios to improve the applicability and scope of application of the trained image segmentation model.

[0121] Step 402: perform image annotation processing on the training image data to obtain training sample data, and perform data enhancement processing on the training sample data to obtain target training data.

[0122] In the disclosed embodiment, when conducting model training, after preparing the data set, the data set is annotated, and screenshots of daily non-WEB interfaces (i.e., the training image data) are collected, and then the screenshots are annotated. When performing the image annotation process, it is necessary to annotate each element in the training image data, for example: classify and annotate the elements such as the input box, title, table, header, table body, etc. on the screenshot (training image data terminal) to form a sample, i.e., the training sample data.

[0123] In order to increase the number and generalization of training sample data, data enhancement processing is required. The methods of data enhancement include but are not limited to: image flipping, contrast adjustment, random adjustment of hue values, etc. Specifically, when performing data enhancement processing, the following methods can be used but are not limited to: scaling the training sample data to the same pixels and performing image enhancement to increase the number and generalization of samples, such as: image flipping, contrast adjustment, random adjustment of hue values ​​and other combined operations.

[0124] Step 403: input the target training data into the image segmentation model for model training processing to obtain the trained image segmentation model, wherein the image segmentation model is a pre-set deep learning model.

[0125] In the disclosed embodiment, target training data is input into the image segmentation model for model training to obtain a trained image segmentation model.

[0126] In one implementable manner of the embodiment of the present disclosure, when the target training data is input into the image segmentation model for model training processing, the following manner may also be adopted but is not limited to: based on the image segmentation model, the target training data is subjected to data conversion processing to obtain a first training vector and a label vector corresponding to the target training data; wherein the image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers; based on the plurality of convolutional layers and the plurality of pooling layers, the first training vector is subjected to vector operation processing to obtain a plurality of second training vectors, and the plurality of second training vectors are subjected to vector concatenation processing to obtain a third training vector; the third training vector is subjected to vector convolution processing, and the third training vector after the convolution processing is subjected to data activation processing through a preset activation function to obtain a fourth training vector; the fourth training vector is subjected to bilinear interpolation processing to obtain a target training vector, and the target training vector and the label vector are subjected to data calculation processing through a preset loss function to obtain a target loss value; the above steps are repeated until the target loss value is less than a preset loss threshold, thereby obtaining the trained image segmentation model.

[0127] In the embodiment of the present disclosure, the loss function is a custom-set function, which can be specifically expressed by, but not limited to, formula (1):

[0128] Loss=-1 / n∑ i t[i]*log(o[i])+(1-t[i]) / log(1-o[i]) formula (1)

[0129] Among them, represents the size of the sample set, t[i] represents the true value of the ith pixel, that is, the label vector, and o[i] is the output value corresponding to the ith pixel of the final output vector, that is, the target training vector.

[0130] In one implementable manner of the disclosed embodiment, after obtaining the matching result, there will be two results, namely, the target text data matches the keyword and the target text data does not match the keyword. At the same time, different processing will be performed according to different matching results. Specifically, after obtaining the matching result, the following method may also be used but is not limited to: when it is determined that the target text data does not match the keyword according to the matching result, the first image data corresponding to the target text data is determined to be an error element.

[0131] In one possible implementation of the embodiment of the present disclosure, in order to facilitate understanding of the implementation process of the embodiment of the present disclosure, reference may be made to Figure 5 , Figure 5An overall processing logic diagram of an element capture method is provided for an embodiment of the present disclosure, wherein an image segmentation model needs to be trained first through a deep learning neural network, and then the interface is segmented into image instances, and the segmented legends are positioned by secondary matching using the Tesserac OCR recognition tool.

[0132] In summary, the embodiments of the present disclosure can achieve the following technical effects:

[0133] The disclosed embodiment uses a deep learning neural network (a trained image segmentation model) to perform image instance segmentation on the image data to be processed, extracts text from the segmented legend, and locates the predetermined elements to be captured by performing secondary matching based on the extracted text, which can reduce the pixels involved. At the same time, the elements to be captured are determined by combining image instance segmentation and text extraction to improve the recognition efficiency and accuracy when capturing elements.

[0134] Corresponding to the above-mentioned method for grasping elements, the present invention also provides a device for grasping elements. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be described in detail in the present invention.

[0135] Figure 6 A schematic diagram of a structure of an element grabbing device provided by an embodiment of the present disclosure, such as Figure 6 As shown, including:

[0136] The segmentation unit 61 is used to input the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data; wherein the image data to be processed includes predetermined elements to be captured;

[0137] An extraction unit 62, configured to perform text extraction processing on the plurality of first image data to obtain target text data corresponding to each of the plurality of first image data;

[0138] A matching unit 63 is used to match the target text data from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the element to be captured;

[0139] A determination unit 64, configured to determine that the first image data corresponding to the target text data is a target element when it is determined according to the matching result that the target text data matches the keyword;

[0140] The encapsulation unit 65 is used to perform object attribute encapsulation processing on the target element to obtain an encapsulated target element.

[0141] The device for element capture provided by the present disclosure inputs the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data; wherein the image data to be processed includes predetermined elements to be captured; performs text extraction processing on the plurality of first image data to obtain target text data corresponding to each of the plurality of first image data; performs matching processing on the target text data from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the elements to be captured; when it is determined according to the matching result that the target text data matches the keyword, the first image data corresponding to the target text data is determined to be the target element, and performs object attribute encapsulation processing on the target element to obtain the encapsulated target element. Compared with the related art, the embodiment of the present disclosure performs image instance segmentation on the image data to be processed through a trained image segmentation model, extracts text from the segmented legend, and locates the predetermined elements to be captured by performing secondary matching according to the extracted text, which can reduce the pixels involved. At the same time, the elements to be captured are determined by combining image instance segmentation and text extraction, which can improve the recognition efficiency and accuracy of element capture.

[0142] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 7 As shown, the segmentation unit 61 includes:

[0143] The conversion module 611 is used to perform data conversion processing on the image data to be processed based on the trained image segmentation model to obtain a first pixel vector corresponding to the image data to be processed; wherein the trained image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers;

[0144] A processing module 612, configured to perform vector operation processing on the first pixel vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second pixel vectors;

[0145] The processing module 612 is further configured to perform vector concatenation processing on the plurality of second pixel vectors to obtain a third pixel vector;

[0146] The processing module 612 is further configured to perform vector convolution processing on the third pixel vector;

[0147] An activation module 613, configured to perform data activation processing on the third pixel vector after the convolution processing by using a preset activation function to obtain a fourth pixel vector;

[0148] The processing module 612 is further configured to perform bilinear interpolation processing on the fourth pixel vector to obtain a target pixel vector, and perform data conversion processing on the target pixel vector to obtain the plurality of first image data.

[0149] Furthermore, in a possible implementation of the embodiment of the present disclosure, the processing module 612 is further configured to:

[0150] Performing a first convolution process on the first pixel vector to obtain a first convolution vector, and performing a first pooling process on the first convolution vector to obtain a first processing vector;

[0151] Performing a second convolution process on the first convolution vector to obtain a second convolution vector, and performing a second pooling process on the second convolution vector to obtain a second processed vector;

[0152] Performing a third convolution process on the second convolution vector to obtain a third convolution vector, and performing a fourth convolution process on the third convolution vector to obtain a third processed vector;

[0153] The third convolution vector is subjected to a fifth convolution process to obtain a fourth convolution vector, and the fourth convolution vector is subjected to a bilinear upsampling process to obtain a fourth processing vector; wherein the plurality of second pixel vectors include at least a first processing vector, a second processing vector, a third processing vector, and a fourth processing vector.

[0154] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 7 As shown, the device also includes:

[0155] An acquisition unit 66 is used to determine an application scenario corresponding to the trained image segmentation model, and acquire a preset amount of training image data based on the application scenario; wherein different application scenarios correspond to different training image data;

[0156] A labeling unit 67, used to perform image labeling processing on the training image data to obtain training sample data;

[0157] An enhancement unit 68, configured to perform data enhancement processing on the training sample data to obtain target training data;

[0158] The training unit 69 is used to input the target training data into the image segmentation model for model training processing to obtain the trained image segmentation model, wherein the image segmentation model is a pre-set deep learning model.

[0159] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 7 As shown, the training unit 69 includes:

[0160] A conversion module 691 is used to perform data conversion processing on the target training data based on the image segmentation model to obtain a first training vector and a label vector corresponding to the target training data; wherein the image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers;

[0161] A processing module 692 is used to perform vector operation processing on the first training vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second training vectors, and perform vector concatenation processing on the multiple second training vectors to obtain a third training vector;

[0162] The processing module 692 is further used to perform vector convolution processing on the third training vector, and perform data activation processing on the third training vector after the convolution processing through a preset activation function to obtain a fourth training vector;

[0163] The processing module 692 is further used to perform bilinear interpolation processing on the fourth training vector to obtain a target training vector;

[0164] A calculation module 693 is used to perform data calculation processing on the target training vector and the label vector by using a preset loss function to obtain a target loss value;

[0165] The repetition module 694 is used to repeat the above steps until the target loss value is less than a preset loss threshold, thereby obtaining the trained image segmentation model.

[0166] Furthermore, in a possible implementation of the embodiment of the present disclosure, the determination unit 64 is also used to determine that the first image data corresponding to the target text data is an error element when it is determined according to the matching result that the target text data does not match the keyword.

[0167] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principle is the same, which is no longer limited in the embodiment of the present disclosure.

[0168] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0169] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0170] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 802 or a computer program loaded from a storage unit 808 to a RAM (Random Access Memory) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An I / O (Input / Output) interface 805 is also connected to the bus 804.

[0171] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0172] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the method of element grabbing. For example, in some embodiments, the method of element grabbing may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the aforementioned element grabbing method in any other appropriate manner (for example, by means of firmware).

[0173] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a special purpose or general purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0175] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0177] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0178] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0179] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0180] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0181] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for element grabbing, characterized in that: include: Inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data; wherein the image data to be processed includes predetermined elements to be captured; Performing text extraction processing on the plurality of first image data to obtain target text data corresponding to each of the plurality of first image data; The target text data is matched from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the element to be captured; When it is determined according to the matching result that the target text data matches the keyword, the first image data corresponding to the target text data is determined as a target element, and object attribute encapsulation processing is performed on the target element to obtain an encapsulated target element.

2. The method according to claim 1, characterized in that The step of inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data includes: Based on the trained image segmentation model, data conversion processing is performed on the image data to be processed to obtain a first pixel vector corresponding to the image data to be processed; wherein the trained image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers; Performing vector operation processing on the first pixel vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second pixel vectors, and performing vector splicing processing on the multiple second pixel vectors to obtain a third pixel vector; Performing vector convolution processing on the third pixel vector, and performing data activation processing on the third pixel vector after the convolution processing by using a preset activation function to obtain a fourth pixel vector; The fourth pixel vector is subjected to bilinear interpolation processing to obtain a target pixel vector, and the target pixel vector is subjected to data conversion processing to obtain the plurality of first image data.

3. The method according to claim 2, characterized in that The performing vector operation processing on the first pixel vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second pixel vectors includes: Performing a first convolution process on the first pixel vector to obtain a first convolution vector, and performing a first pooling process on the first convolution vector to obtain a first processing vector; Performing a second convolution process on the first convolution vector to obtain a second convolution vector, and performing a second pooling process on the second convolution vector to obtain a second processed vector; Performing a third convolution process on the second convolution vector to obtain a third convolution vector, and performing a fourth convolution process on the third convolution vector to obtain a third processed vector; The third convolution vector is subjected to a fifth convolution process to obtain a fourth convolution vector, and the fourth convolution vector is subjected to a bilinear upsampling process to obtain a fourth processing vector; wherein the plurality of second pixel vectors include at least a first processing vector, a second processing vector, a third processing vector, and a fourth processing vector.

4. The method according to claim 1, characterized in that: Before inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data, the method further includes: Determine an application scenario corresponding to the trained image segmentation model, and obtain a preset amount of training image data based on the application scenario; wherein different application scenarios correspond to different training image data; Performing image annotation processing on the training image data to obtain training sample data, and performing data enhancement processing on the training sample data to obtain target training data; The target training data is input into the image segmentation model for model training processing to obtain the trained image segmentation model, wherein the image segmentation model is a pre-set deep learning model.

5. The method according to claim 4, characterized in that The step of inputting the target training data into the image segmentation model for model training to obtain the trained image segmentation model comprises: Based on the image segmentation model, the target training data is subjected to data conversion processing to obtain a first training vector and a label vector corresponding to the target training data; wherein the image segmentation model includes at least a plurality of convolutional layers and a plurality of pooling layers; Performing vector operation processing on the first training vector based on the multiple convolutional layers and the multiple pooling layers to obtain multiple second training vectors, and performing vector concatenation processing on the multiple second training vectors to obtain a third training vector; Performing vector convolution processing on the third training vector, and performing data activation processing on the third training vector after the convolution processing by using a preset activation function to obtain a fourth training vector; Performing bilinear interpolation processing on the fourth training vector to obtain a target training vector, and performing data calculation processing on the target training vector and the label vector using a preset loss function to obtain a target loss value; Repeat the above steps until the target loss value is less than the preset loss threshold, and then obtain the trained image segmentation model.

6. The method according to claim 1, characterized in that After matching the target text data from a preset keyword database to obtain a matching result, the method further includes: When it is determined according to the matching result that the target text data does not match the keyword, it is determined that the first image data corresponding to the target text data is an error element.

7. A device for grasping an element, characterized in that: include: A segmentation unit, used for inputting the acquired image data to be processed into a trained image segmentation model for image segmentation processing to obtain a plurality of first image data; wherein the image data to be processed includes predetermined elements to be captured; An extraction unit, configured to perform text extraction processing on the plurality of first image data to obtain target text data corresponding to each of the plurality of first image data; A matching unit, used for matching the target text data from a preset keyword database to obtain a matching result; wherein the preset keyword database is a database containing keywords corresponding to the elements to be captured; a determining unit, configured to determine, when it is determined according to the matching result that the target text data matches the keyword, that the first image data corresponding to the target text data is a target element; The encapsulation unit is used to perform object attribute encapsulation processing on the target element to obtain the encapsulated target element.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images