Information processing method, device, readable storage medium and electronic device

Abstract

The present invention discloses an information processing method, device, readable storage medium and electronic device. The method comprises: determining a target object to be matched on a graphical user interface; determining a target area where the target object is located on the graphical user interface; performing regression processing on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface; matching the target object based on the regression result to obtain a matching result. The present invention solves the technical problem of low accuracy of object matching.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an information processing method, device, readable storage medium and electronic device. Background Art

[0002] At present, when matching objects in games, image matching algorithms are usually used. However, due to the limitations of the image matching algorithms themselves and the limitations of automated scenarios, the accuracy of the image matching algorithms is difficult to reach the standard acceptable to users, resulting in a technical problem of low accuracy of object matching.

[0003] Currently, no effective solution has been proposed for the above-mentioned technical problem of low accuracy in object matching. Summary of the invention

[0004] At least some embodiments of the present invention provide an information processing method, apparatus, readable storage medium and electronic device to at least solve the technical problem of low accuracy of object matching.

[0005] According to one embodiment of the present invention, there is provided an information processing method, the method comprising: determining a target object to be matched on a graphical user interface; determining a target area where the target object is located on the graphical user interface; performing regression processing on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface; matching the target object based on the regression result to obtain a matching result.

[0006] Optionally, determining the target area where the target object is located on the graphical user interface includes: acquiring an original bounding box of the target object; and determining the target area where the target object is located on the graphical user interface using the original bounding box.

[0007] Optionally, performing regression processing on the target area to obtain a regression result includes: performing regression processing on the original bounding box based on a regression model to obtain a target bounding box of the target object, wherein the regression model is a model determined based on a loss function and used to perform regression processing on the input bounding box.

[0008] Optionally, regressing the original bounding box based on the regression model to obtain a target bounding box of the target object includes: extracting features of the target object from the original image of the target object; and regressing the original bounding box and the features based on the regression model to obtain the target bounding box.

[0009] Optionally, in response to a first request from the client, the regression model is queried.

[0010] Optionally, in response to a first request from a client, querying a regression model includes: identifying a query parameter carried in the first request, and querying a regression model that matches the query parameter.

[0011] Optionally, in response to a second request from the client, the regression model is updated.

[0012] Optionally, in response to the second request from the client, updating the regression model includes: identifying update data carried in the second request, and updating the regression model based on the update data.

[0013] Optionally, the loss function is obtained based on a target parameter, and the target parameter is used to make the regression model converge.

[0014] Optionally, the original model is converted into a target model, and the target model is adjusted based on the loss function to obtain a regression model.

[0015] Optionally, determining the target object to be matched on the graphical user interface includes: determining an original object to be matched on the graphical user interface; and preprocessing the original object to obtain the target object.

[0016] Optionally, preprocessing the original object to obtain the target object includes: performing data enhancement on the original object to obtain the target object.

[0017] Optionally, the matching result is post-processed to obtain a post-processing result; and the post-processing result is returned to the client.

[0018] Optionally, matching the target object based on the regression result to obtain a matching result includes: searching the target image for a matching object whose similarity with the target object is greater than a target threshold based on the regression result; and determining the matching object as the matching result.

[0019] According to one embodiment of the present invention, there is also provided an information processing device, which may include: a first determination unit, used to determine a target object to be matched on a graphical user interface; a second determination unit, used to determine a target area where the target object is located on the graphical user interface; a regression unit, used to perform regression processing on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface; and a matching unit, used to match the target object based on the regression result to obtain a matching result.

[0020] In order to achieve the above object, according to another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the information processing method of the embodiment of the present invention.

[0021] To achieve the above object, according to another aspect of the present invention, an electronic device is provided. The electronic device may include a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the computer program to execute the information processing method of the embodiment of the present invention.

[0022] In at least some embodiments of the present invention, a target object to be matched on a graphical user interface is determined; a target area where the target object is located on the graphical user interface is determined; a regression process is performed on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface; and the target object is matched based on the regression result to obtain a matching result. In other words, the present invention locates the target object on the graphical user interface by using a regression algorithm, and then outputs a precise target area, and uses it as the input of an image matching algorithm. The algorithm can be connected to any image matching algorithm to achieve the purpose of reducing the limitations of the image matching algorithm, thereby achieving the technical effect of improving the accuracy of object matching, and then solving the technical problem of low accuracy of object matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 is a hardware structure block diagram of a mobile terminal according to an information processing method of one embodiment of the present invention;

[0025] Figure 2 is a flow chart of an information processing method according to one embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of an adaptive user interface field of view algorithm in an automated testing scenario according to an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of a high-performance online model deployment solution according to an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of an information processing effect according to an embodiment of the present invention;

[0029] Figure 6 is a structural block diagram of an information processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] According to one embodiment of the present invention, an embodiment of an information processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] The method embodiment can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, the mobile terminal can be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID for short), a PAD, a game console and other terminal devices. Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of an information processing method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microprocessor (MCU), a programmable logic device (FPGA), a neural network processor (NPU), a tensor processor (TPU), an artificial intelligence (AI) type processor, etc.) and a memory 104 for storing data. Optionally, the mobile terminal may also include a transmission device 106 for communication functions, an input and output device 108, and a display device 110. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0034] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the information processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above-mentioned information processing method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] The inputs in the input / output device 108 may come from a plurality of human interface devices (HIDs), such as keyboards and mice, game controllers, and other dedicated game controllers (such as steering wheels, fishing rods, dance mats, remote controls, etc.). In addition to providing input functions, some human interface devices may also provide output functions, such as force feedback and vibration of game controllers, audio output of controllers, etc.

[0037] The display device 110 may be, for example, a head-up display (HUD), a touch-screen liquid crystal display (LCD), and a touch display (also referred to as a "touch screen" or "touch display screen"). The liquid crystal display may enable a user to interact with a user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), and a user may interact with the GUI by finger contacts and / or gestures on a touch-sensitive surface, wherein the human-computer interaction functions here may optionally include the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital videos, playing digital music, and / or web browsing, etc. The executable instructions for executing the above-mentioned human-computer interaction functions are configured / stored in a computer program product or a readable storage medium executable by one or more processors.

[0038] In this embodiment, an information processing method running on the above mobile terminal is provided. Figure 2 FIG. 1 is a flow chart of an information processing method according to one embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0039] Step S202: determining a target object to be matched on the graphical user interface.

[0040] In the technical solution provided in the above step S202 of the present invention, it is determined that the target object to be matched is on the graphical user interface, wherein the graphical user interface may be an operation interface displayed by the user client, and the target object may be a functional control on the graphical user interface, such as an operation button.

[0041] Optionally, the target object to be matched on the graphical user interface can be obtained by preprocessing the original object in the middleware of the server, or by performing data enhancement on the original object, wherein the original object can be an image on the graphical user interface of the user client.

[0042] Step S204: determining a target area where the target object is located on the graphical user interface.

[0043] In the technical solution provided in the above step S204 of the present invention, the target object is positioned on the graphical user interface to obtain a target area where the target object is located on the graphical user interface, wherein the target area can be the area occupied by the position of the target object on the graphical user interface, that is, the area occupied by the position of the operation button on the graphical user interface.

[0044] Optionally, the (x, y, w, h) tuple can be used in a regression model to locate the target object and serve as the network learning / regression target.

[0045] Step S206 , performing regression processing on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface.

[0046] In the technical solution provided in the above step S206 of the present invention, a regression model is trained to perform regression processing on the target area, and the display area of ​​the target object on the graphical user interface is located based on the obtained regression result, wherein the regression result can be a normalized value of the target area, and the display area can be the field of view of the graphical user interface.

[0047] Optionally, this embodiment can utilize the idea of ​​bounding box regression in target detection and eliminate the classification branch in target detection, so as to directly regress the graphical user interface field of view, wherein the bounding box regression means that in order to detect the position of the target (expressed as a bounding box), it is necessary to perform a regression on the bounding box.

[0048] For example, in the regression model, the image width (w), height (h) and logarithm (log) are used as constraints to accelerate convergence, obtain the coordinates of the bounding box, and then output the normalized coordinate values ​​(tx`, ty`, tw`, th`). The graphical user interface field of view can be positioned based on the normalized coordinate values, where tx` is the horizontal coordinate of the center point of the bounding box after regression, ty` is the vertical coordinate of the center point of the bounding box after regression, tw` is the width of the bounding box after regression, and th` is the height of the bounding box after regression.

[0049] Step S208: matching the target object based on the regression result to obtain a matching result.

[0050] In the technical solution provided in the above step S208 of the present invention, after the target area is regressed, a regression result is obtained, and the regression result is used as an input of the matching algorithm to match the target object, thereby obtaining a matching result.

[0051] Optionally, the regression result obtained by the algorithm proposed in the present invention can be connected to any unsupervised image matching algorithm (such as a deep learning algorithm, or a traditional scale-invariant feature transformation algorithm), which can greatly improve the object matching accuracy.

[0052] Optionally, when the client performs image preprocessing operations, it will cause lag in the automated testing tool. In order to avoid lag, this embodiment builds a high-stability, high-concurrency and scalable online model deployment solution with server (flask) + framework (nginx) + protocol (uwsgi) as middleware and machine learning model online deployment service framework (tensorflow serving, abbreviated as tfs) to provide model reasoning services to solve the technical problem of low accuracy of object matching.

[0053] Through the above steps S202 to S208 of the present application, the target object to be matched on the graphical user interface is determined; the target area where the target object is located on the graphical user interface is determined; regression processing is performed on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface; the target object is matched based on the regression result to obtain a matching result. In other words, the present invention locates the target object on the graphical user interface by using a regression algorithm, and then outputs a precise target area, and uses it as the input of the image matching algorithm. The algorithm can be connected to any image matching algorithm to achieve the purpose of reducing the limitations of the image matching algorithm, thereby achieving the technical effect of improving the accuracy of object matching, and then solving the technical problem of low accuracy of object matching.

[0054] The above method of this embodiment is further introduced below.

[0055] As an optional implementation, step S204, determining the target area where the target object is located on the graphical user interface, includes: obtaining an original bounding box of the target object; and determining the target area where the target object is located on the graphical user interface using the original bounding box.

[0056] In this embodiment, the original bounding box of the target object is obtained, and the target area where the target object is located on the graphical user interface is determined using the original bounding box, that is, the position of the target object on the graphical user interface is determined using the original bounding box of the target object, wherein the original bounding box may be a bounding box before regression is performed.

[0057] For example, referring to the numerical normalization scheme in the classic target detection algorithm (faster-rcnn), the image width (w), height (h) and logarithm (log) are used as constraints to accelerate convergence. The coordinates of the bounding box BBOX (t x , ty , t w , t h ), where tx is the horizontal coordinate of the center point of the bounding box, ty is the vertical coordinate of the center point of the bounding box, tw is the width of the bounding box, and th is the height of the bounding box.

[0058] t x =(g x -p x ) / p w

[0059] t y =(g y -p y ) / p h

[0060] t w =log(g w / p w )

[0061] t h =log(g h / p h )

[0062] Among them, g x , g y , g w , g h and p x , p y , p w , p h For x , t y , t w , t h The corresponding mapping coordinates.

[0063] As an optional implementation, step S206, regressing the target area to obtain a regression result, includes: regressing the original bounding box based on a regression model to obtain a target bounding box of the target object, wherein the regression model is determined based on a loss function and is used to regress the input bounding box.

[0064] In this embodiment, a regression model is determined based on a loss function, and the original bounding box is regressed using the regression model to obtain a target bounding box of the target object, wherein the target bounding box may be a bounding box after regression, and the target bounding box of the target object may be used to determine the graphical user interface field of view of the target object.

[0065] Optionally, the loss function may be a target detection regression loss function (Complete-IOU Loss), which adds an influencing factor to enable the network to converge faster while achieving better performance, thereby avoiding the situation where the gradient disappears when the intersection over Union (IOU) of two bounding boxes does not intersect. The influencing factor may be a factor in the target detection regression loss function, which is a parameter used to measure the consistency of the aspect ratio, and may allow the network to converge faster.

[0066] As an optional implementation, regressing the original bounding box based on a regression model to obtain a target bounding box of the target object includes: extracting features of the target object from the original image of the target object; and regressing the original bounding box and the features based on the regression model to obtain the target bounding box.

[0067] In this embodiment, a lightweight convolutional neural network (InceptionV3) is used as the backbone network to extract features of the target object from the original image of the target object, which are also image features, and the extracted features are added to the feature pyramid (FPN). Then, the original bounding box and the features are regressed based on the regression model to obtain the target bounding box, wherein the numerical normalization scheme in the target detection algorithm (faster-rcnn) can be referred to, and the width (w), height (h) and other information of the target object determined by the image features are used as constraints to accelerate convergence, and the normalized target bounding box is output to reduce the waiting time of a single click in the automation process and alleviate the scale change of the target object graphical user interface.

[0068] It should be noted that the original bounding box of this embodiment has no features, and the features processed by this embodiment are image features of the original image of the target object.

[0069] As an optional implementation, in response to the first request from the client, the regression model is queried.

[0070] In this embodiment, the server responds to the first request from the client and queries the corresponding regression model according to the first request, that is, the server is equipped with a regression model query service to facilitate users to request regression models on demand, wherein the first request can be a request from the user client for a specific regression model and version number.

[0071] As an optional implementation, in response to a first request from a client, querying a regression model includes: identifying a query parameter carried in the first request, and querying a regression model that matches the query parameter.

[0072] In this embodiment, when the client initiates a first request to the server, the server identifies the query parameters carried in the first request, and determines a regression model that matches the query parameters based on the query parameters, wherein the query parameters may be parameters used to distinguish regression models, such as a version number.

[0073] As an optional implementation, in response to the second request from the client, the regression model is updated.

[0074] In this embodiment, the regression model can be updated online. The server responds to the second request initiated by the client, and determines the update data that matches the second request based on the relevant parameters of the regression model in the second request, and uses the updated data to update the regression model online, wherein the second request can be an update request for the regression model, including the relevant parameters of the regression model to be updated.

[0075] Optionally, the second request can also be a request script for the online hot update framework (grpc) of the machine learning model online deployment service framework (tensorflowserving, abbreviated as tfs).

[0076] Optionally, the update data provided by the server to the client may be the latest version data of the regression model, or may be update patch data of the regression model, which is not specifically limited here.

[0077] Optionally, the regression model is updated online, and the current service of the regression model does not need to be stopped during the update.

[0078] As an optional implementation, in response to the second request from the client, updating the regression model includes: identifying update data carried in the second request, and updating the regression model based on the update data.

[0079] In this embodiment, when the client initiates a second request to the server, the server responds to the second request initiated by the client and determines the update data that matches it based on the relevant parameters of the regression model in the second request. For example, the update data is the latest version number and the latest version data packet of the regression model. The server then identifies the update data carried by the second request and updates the regression model based on the update data.

[0080] As an optional implementation, the loss function is obtained based on a target parameter, and the target parameter is used to make the regression model converge.

[0081] In this embodiment, the loss function can be obtained by adding a target parameter, so that the network converges faster and achieves better performance. The target parameter can be an influencing factor, which is a factor in the target detection regression loss function and a parameter used to measure the consistency of the aspect ratio, so that the network can converge faster.

[0082] As an optional implementation, the original model is converted into a target model, and the target model is adjusted based on the loss function to obtain a regression model.

[0083] In this embodiment, the model file is converted with one click, and the original model is converted into a target model. After the conversion, the target model is adjusted based on the loss function to obtain a regression model, wherein the original model can be a trained model and the target model can be a model used online.

[0084] For example, through one-click model conversion, models of three different artificial intelligence development frameworks (tensorflow, keras, pytorch) can be converted into binary files in the format (Saved Model) required by the machine learning model online deployment service framework.

[0085] As an optional implementation, step S202, determining a target object to be matched on the graphical user interface, includes: determining an original object to be matched on the graphical user interface; and preprocessing the original object to obtain the target object.

[0086] In this embodiment, the original object to be matched on the graphical user interface is obtained, and the original object is preprocessed in the server middleware to obtain the target object, so as to avoid client lag caused by client image preprocessing operations, wherein the original object can be an image on the user client graphical user interface.

[0087] As an optional implementation, preprocessing the original object to obtain the target object includes: performing data enhancement on the original object to obtain the target object.

[0088] In this embodiment, data enhancement can be performed on the original object through various data enhancement strategies to obtain the target object.

[0089] Optionally, since the user interfaces in the screens of different models may be offset, scaled, etc., the present invention adopts a variety of data enhancement strategies to implement data enhancement schemes such as rotation, invert, brightness change, and whitening to adapt to samples of different models.

[0090] As an optional implementation, the matching result is post-processed to obtain a post-processing result; and the post-processing result is returned to the client.

[0091] In this embodiment, after regression processing is performed on the target area, a regression result is obtained, and the regression result is used as the input of the matching algorithm to match the target object to obtain a matching result. The matching result is post-processed in the server middleware, and the post-processed result is returned to the client, thereby simplifying the functions of the client and achieving the purpose of avoiding frequent freezes in the client.

[0092] As an optional implementation, step S208, matching the target object based on the regression result to obtain a matching result, includes: searching the target image for a matching object whose similarity with the target object is greater than a target threshold based on the regression result; and determining the matching object as the matching result.

[0093] In this embodiment, a matching object that matches the target object is searched in the target image based on the regression result. When the similarity between the matching object in the target image and the target object is greater than a target threshold, the matching object can be determined as a matching result, wherein the target threshold can be the similarity limit between the matching object and the target object, which can be set according to actual needs.

[0094] The technical solution of the embodiment of the present invention is further introduced below with examples in combination with preferred implementation modes.

[0095] At present, the template matching algorithms used in the industry are generally divided into the following categories: represented by traditional operators such as Scale-Invariant Feature Transform (SIFT) / Speeded Up Robust Features (SURF), which match the number of local invariant feature points, and remove mismatched points with algorithms such as random sample consensus (RANSAC) and Brute Force (BF); deep learning solutions, which extract features using pre-trained models, send them to the Normalized Cross-Correlation (NCC) logic to obtain an integral map and take the maximum feature area, thereby mapping back to the original image to obtain the target area. However, these algorithms cannot achieve a high matching accuracy in automated testing scenarios.

[0096] In related technologies, image recognition (image matching) technology is usually used to automatically perform equipment operations in automated test scenarios to replace manual repetitive operations. However, due to the limitations of image matching algorithms, namely the small effective feature area of ​​user interface (UI) buttons, and the limitations of automated scenarios, namely inaccurate clicks when recording use cases, the accuracy of all image matching algorithms is difficult to reach the standard line acceptable to users.

[0097] Before using the recorder to record use cases and generate automated scripts, you need to manually adjust the preset image size (the length of the preset image is one tenth of the length of the original image) as input for the subsequent image matching algorithm. Otherwise, there will be a large number of matching failures. This is because: when the user records the use case, the click is inaccurate (not the center point but the user interface button is effective). On devices with other resolutions, the center point deviates seriously due to image deformation, resulting in the simulated click on the center point of the region of interest (ROI) but the button is not actually clicked; the effective area of ​​the user interface button is too small (small feature area), and a large number of irrelevant features are introduced, resulting in image matching failure.

[0098] In another related technology, manually adjusting the size of the preset image can greatly improve the accuracy of image matching. However, during the test, it was found that the time cost of manually adjusting the template was too high. For example, in a 30-minute use case, the time spent on manually adjusting the template accounted for more than 70%. In addition, manually adjusting the size of the preset image requires certain screenshot skills to ensure sufficient effective features.

[0099] However, the present invention proposes an adaptive user interface field of view algorithm, which locates the target user interface by training a regression model, outputs the precise user interface position, and uses it as the input of the image matching algorithm. The algorithm can be connected with any image matching algorithm (deep learning algorithm or traditional scale-invariant feature transformation algorithm) to reduce the limitations of traditional image matching algorithms and improve the accuracy of image matching.

[0100] Next, an adaptive user interface field of view algorithm in an automated testing scenario provided by an embodiment of the present invention is further introduced. Figure 3 As shown, Figure 3 is a schematic diagram of an adaptive user interface field of view algorithm in an automated testing scenario according to an embodiment of the present invention, which may include:

[0101] The first part is the input of the original object.

[0102] Since the user interfaces in the screens of different models may be offset, scaled, etc., this embodiment uses a variety of data enhancement strategies to process the original objects and input them into the regression model.

[0103] Optionally, in order to adapt to samples of different models, this embodiment implements data enhancement schemes such as rotation, inverting, brightness change and whitening.

[0104] In the second part, select the backbone network.

[0105] A lightweight convolutional neural network (InceptionV3) is selected as the backbone network to accelerate the reasoning process and reduce the waiting time of a single click in the automation process. Feature Pyramid Networks (FPN) is added to alleviate the problem of scale change of the target user interface.

[0106] The third part is regression.

[0107] The target user interface is located with the (x, y, w, h) tuple as the network learning / regression target. The numerical normalization scheme in the classic target detection algorithm (faster-rcnn) is referred to. The image width (w), height (h) and logarithm (log) are used as constraints to accelerate convergence, and the coordinates of the bounding box are obtained, and then the normalized coordinate values ​​(t x `、t y `、t w `、t h `), where tx` is the horizontal coordinate of the center point of the bounding box after regression, ty` is the vertical coordinate of the center point of the bounding box after regression, tw` is the width of the bounding box after regression, and th` is the height of the bounding box after regression.

[0108] Alternatively, the coordinates of the bounding box BBOX(t x , t y , t w , t h ), where tx is the horizontal coordinate of the center point of the bounding box, ty is the vertical coordinate of the center point of the bounding box, tw is the width of the bounding box, and th is the height of the bounding box.

[0109] t x =(g x -p x ) / p w

[0110] t y =(g y-p y ) / p h

[0111] t w =log(g w / p w )

[0112] t h =log(g h / p h )

[0113] Among them, g x , g y , g w , g h and p x , p y , p w , p h For x , t y , t w , t h The corresponding mapping coordinates. The fourth part is the loss function.

[0114] Using the target detection regression loss function (Complete-IOU Loss) as the loss function, in order to avoid the gradient vanishing when the intersection over Union (IOU) of two bounding boxes does not intersect, the network can be made to converge faster and achieve better performance by adding an influence factor. The influence factor can be a factor in the target detection regression loss function, which is a parameter used to measure the consistency of the aspect ratio, and can make the network converge faster.

[0115] Optionally, the target detection regression loss function is,

[0116]

[0117] Among them, L CIoU can be the target detection regression loss function, IOU can be the intersection-over-union ratio of two bounding boxes, b can be the center point position of the predicted box, and b gt It can be the center point position of the real box, ρ can be the Euclidean distance between the two center points, c can be the diagonal distance of the minimum closure area that can contain both the predicted box and the real box, and αv is the penalty factor.

[0118] In order to avoid the jamming of automated testing tools due to I / O intensive computing (mainly client image preprocessing operations), the present invention also builds a set of online model deployment solutions with high stability, high concurrency and scalability, using server (flask) + framework (nginx) + protocol (uwsgi) as middleware and machine learning model online deployment service framework (tensorflow serving, abbreviated as tfs) to provide model inference services, such as Figure 4 As shown, Figure 4 4 is a schematic diagram of a high-performance online model deployment solution according to an embodiment of the present invention, including: a client 401 and a server middleware 402.

[0119] The server middleware 402 is responsible for image pre-processing and post-processing tasks, and at the same time receives concurrent requests from users and concurrently requests the machine learning model online deployment service framework to obtain model inference results.

[0120] Model files can be converted with one click, and models of three different artificial intelligence development frameworks (tensorflow, keras, pytorch) can be converted into binary files in the format (SavedModel) required by the machine learning model online deployment service framework.

[0121] Multi-model and multi-version control provides the corresponding server with an application programming interface (API) for requesting specific models and version numbers.

[0122] Provides a framework (grpc) request script for online hot update of the machine learning model online deployment service framework.

[0123] Provide an independent online model query service in service, so that users can request models on demand.

[0124] This embodiment uses the above method to verify the effect of dozens of common games. The image matching accuracy can reach more than 98%. The manual intervention caused by image matching failure is reduced by more than 98%. That is, the average intersection of the model output bounding box and the manually adjusted standard bounding box is greater than 98%. Some of the prediction effects are as follows: Figure 5 As shown, Figure 5 It is a schematic diagram of an information processing effect according to an embodiment of the present invention. In the figure, the dotted line represents a manually adjusted standard bounding box, and the solid line represents the bounding box output by the model. It can be seen that the dotted line and the solid line basically coincide with each other, that is, the bounding box output by the model and the manually adjusted standard bounding box basically coincide with each other, which indicates that the accuracy of matching images using the bounding box output by the model is relatively high, and the model can be a regression model.

[0125] The embodiment of the present invention utilizes the idea of ​​bounding box regression in target detection, eliminates the classification branches in target detection, directly regresses the user interface field of view, and builds a set of server + framework + protocol middleware services to avoid the jamming of automated testing tools during client image preprocessing operations, thereby solving the technical problem of low image matching accuracy and achieving the technical effect of improving image matching accuracy.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0127] The embodiment of the present invention further provides an information processing device, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "unit" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0128] Figure 6 is a structural block diagram of an information processing device according to an embodiment of the present invention. Figure 6 As shown, the information processing device 600 may include: a first determination unit 601 , a second determination unit 602 , a regression unit 603 and a matching unit 604 .

[0129] A first determining unit 601 is used to determine a target object to be matched on a graphical user interface;

[0130] A second determining unit 602 is used to determine a target area where a target object is located on the graphical user interface;

[0131] A regression unit 603 is used to perform regression processing on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface;

[0132] The matching unit 604 is used to match the target object based on the regression result to obtain a matching result.

[0133] Optionally, the second determining unit 602 includes: an acquiring module, configured to acquire an original bounding box of the target object; and a determining module, configured to determine a target area where the target object is located on the graphical user interface using the original bounding box.

[0134] Optionally, the regression unit 603 includes: a processing module, which is used to perform regression processing on the original bounding box based on a regression model to obtain a target bounding box of the target object, wherein the regression model is a model determined based on a loss function and used to perform regression processing on the input bounding box.

[0135] Optionally, the processing module includes: an extraction submodule, used to extract features of the target object from the original image of the target object; and a processing submodule, used to perform regression processing on the original bounding box and the features based on a regression model to obtain a target bounding box.

[0136] Optionally, the device may further include: a first processing unit, configured to respond to a first request from a client and query the regression model.

[0137] Optionally, the first processing unit includes: a first processing module, configured to identify a query parameter carried in the first request, and query a regression model matching the query parameter.

[0138] Optionally, the device may further include: a second processing unit, configured to update the regression model in response to a second request from the client.

[0139] Optionally, the second processing unit includes: a second processing module, configured to identify update data carried by the second request, and update the regression model based on the update data.

[0140] Optionally, the loss function is obtained based on a target parameter, and the target parameter is used to make the regression model converge.

[0141] Optionally, the device may further include: a third processing unit, configured to convert the original model into a target model, and adjust the target model based on a loss function to obtain a regression model.

[0142] Optionally, the first determining unit 601 includes: a first determining module, used to determine an original object to be matched on the graphical user interface; and a first preprocessing module, used to preprocess the original object to obtain a target object.

[0143] Optionally, the first preprocessing module is used to preprocess the original object through the following steps to obtain the target object: perform data enhancement on the original object to obtain the target object.

[0144] Optionally, the device may further include: a fourth processing unit, configured to post-process the matching result to obtain a post-processing result; and a returning unit, configured to return the post-processing result to the client.

[0145] Optionally, the matching unit 604 includes: a search module, configured to search the target image for a matching object whose similarity with the target object is greater than a target threshold based on the regression result; and a second determination module, configured to determine the matching object as a matching result.

[0146] In the information processing device of this embodiment, a target object to be matched on a graphical user interface is determined by a first determination unit; a second determination unit determines a target area where the target object is located on the graphical user interface; a regression unit performs regression processing on the target area to obtain a regression result; and a matching unit matches the target object based on the regression result to obtain a matching result, thereby achieving a technical effect of improving the accuracy of object matching and further solving the technical problem of low accuracy of object matching.

[0147] It should be noted that the above-mentioned units and modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above-mentioned units and modules are all located in the same processor; or, the above-mentioned units and modules are located in different processors in any combination.

[0148] An embodiment of the present invention further provides a non-volatile storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0149] Optionally, in this embodiment, the non-volatile storage medium may be configured to store a computer program for performing the following steps:

[0150] S1, determining a target object to be matched on a graphical user interface;

[0151] S2, determining a target area where the target object is located on the graphical user interface;

[0152] S3, performing regression processing on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface;

[0153] S4, matching the target object based on the regression result to obtain a matching result.

[0154] Optionally, in this embodiment, the above-mentioned non-volatile storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0155] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0156] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0157] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0158] S1, determining a target object to be matched on a graphical user interface;

[0159] S2, determining a target area where the target object is located on the graphical user interface;

[0160] S3, performing regression processing on the target area to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface;

[0161] S4, matching the target object based on the regression result to obtain a matching result.

[0162] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0163] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0164] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0167] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0169] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An information processing method, characterized in that: include: determining a target object to be matched on the graphical user interface; Determine a target area where the target object is located on the graphical user interface; The target area is regressed based on a regression model to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface, and the regression model is determined based on a loss function, and the loss function is L CIoU is used to represent the loss function, IoU is used to represent the intersection over union ratio of the input bounding box, b is used to represent the position of the center point of the predicted box, and b gt is used to represent the position of the center point of the real box, ρ is used to represent the Euclidean distance between the center point of the predicted box and the center point of the real box, c is used to represent the diagonal distance of the minimum closure area including the predicted box and the real box, and αν is used to represent the penalty factor of the loss function; The target object is matched based on the regression result to obtain a matching result.

2. The method according to claim 1, characterized in that: Determining a target area where the target object is located on the graphical user interface includes: Obtaining an original bounding box of the target object; The target area where the target object is located on the graphical user interface is determined using the original bounding box.

3. The method according to claim 2, characterized in that Performing regression processing on the target area based on the regression model to obtain a regression result includes: The original bounding box is regressed based on the regression model to obtain a target bounding box of the target object, wherein the regression model is a model for performing regression processing on the input bounding box.

4. The method according to claim 3, characterized in that Performing regression processing on the original bounding box based on the regression model to obtain a target bounding box of the target object includes: Extracting features of the target object from an original image of the target object; The original bounding box and the feature are regressed based on the regression model to obtain the target bounding box.

5. The method according to claim 3, characterized in that: The method further comprises: In response to a first request from a client, the regression model is queried.

6. The method according to claim 5, characterized in that In response to a first request from a client, querying the regression model includes: A query parameter carried in the first request is identified, and the regression model matching the query parameter is searched.

7. The method according to claim 3, characterized in that The method further comprises: In response to a second request from the client, the regression model is updated.

8. The method according to claim 7, characterized in that In response to the second request from the client, updating the regression model includes: The update data carried by the second request is identified, and the regression model is updated based on the update data.

9. The method according to claim 3, characterized in that: The loss function is obtained based on a target parameter, and the target parameter is used to make the regression model converge.

10. The method according to claim 3, characterized in that The method further comprises: The original model is converted into a target model, and the target model is adjusted based on the loss function to obtain the regression model.

11. The method according to any one of claims 1 to 10, characterized in that Determine the target object to be matched on the graphical user interface, including: Determining an original object to be matched on the graphical user interface; The original object is preprocessed to obtain the target object.

12. The method according to claim 11, characterized in that Preprocessing the original object to obtain the target object includes: Perform data enhancement on the original object to obtain the target object.

13. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Post-processing the matching result to obtain a post-processing result; The post-processing result is returned to the client.

14. The method according to any one of claims 1 to 10, characterized in that Matching the target object based on the regression result to obtain a matching result includes: Based on the regression result, searching in the target image for a matching object whose similarity with the target object is greater than a target threshold; The matching object is determined as the matching result.

15. An information processing device, characterized in that: include: A first determining unit, used to determine a target object to be matched on the graphical user interface; A second determining unit, configured to determine a target area where the target object is located on the graphical user interface; A regression unit is used to perform regression processing on the target area based on a regression model to obtain a regression result, wherein the regression result is used to locate the display area of ​​the target object on the graphical user interface, the regression model is determined based on a loss function, and the loss function is L CIoU is used to represent the loss function, IoU is used to represent the intersection over union ratio of the input bounding box, b is used to represent the position of the center point of the predicted box, and b gt is used to represent the position of the center point of the real box, ρ is used to represent the Euclidean distance between the center point of the predicted box and the center point of the real box, c is used to represent the diagonal distance of the minimum closure area including the predicted box and the real box, and αν is used to represent the penalty factor of the loss function; A matching unit is used to match the target object based on the regression result to obtain a matching result.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 14 when executed by a processor.

17. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method described in any one of claims 1 to 14.

Images