Device operation guide file generation method, device, electronic device and storage medium

Abstract

The present invention relates to artificial intelligence technology, and discloses a method for generating a device operation guide file, comprising: performing operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot set to obtain multiple classification screenshot sets; performing image area division and annotation on each operation interface screenshot in the classification screenshot set to obtain annotated screenshots; performing image content description on the annotated screenshots to obtain image content text; using each image content text to annotate the corresponding operation interface screenshot; combining and packaging all the annotated operation interface screenshots in the order of the corresponding screenshot time to obtain an operation guide file. The present invention also relates to a blockchain technology, and the operation interface screenshot set can be stored in a blockchain node. The present invention also proposes a device, device and medium for generating a device operation guide file. The present invention can improve the efficiency of generating device operation guide files.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for generating a device operation guide file. Background Art

[0002] In recent years, with the continuous development of artificial intelligence technology, bank branches have realized self-service business processing for users through various devices. However, the operation of some devices is not very user-friendly, and the software version of the device is often updated iteratively, so the operation guide of the device is very necessary for users.

[0003] However, the traditional device operation guide generation solution requires screenshot description of all operation screenshots of all devices, and the efficiency of the device operation guide generation method is low. Summary of the invention

[0004] The present invention provides a method and device for generating a device operation guide file, an electronic device and a computer-readable storage medium, the main purpose of which is to improve the efficiency of generating the device operation guide file.

[0005] To achieve the above object, the present invention provides a method for generating a device operation guide file, comprising:

[0006] Acquire an operation interface screenshot set, wherein the operation interface screenshot set includes multiple operation interface screenshots, and each operation interface screenshot includes a corresponding screenshot time;

[0007] Classifying the operation interface screenshots in the operation interface screenshot set and filtering similar screenshots to obtain a plurality of classified screenshot sets;

[0008] Performing image region division and annotation on each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot;

[0009] Describing the image content of the annotated screenshot to obtain image content text;

[0010] Annotate the corresponding operation interface screenshot with the text of each image content;

[0011] All the annotated operation interface screenshots are combined and packaged in the order of the corresponding screenshot time to obtain an operation guide file.

[0012] Optionally, the operation interface screenshots in the operation interface screenshot set are classified and similar screenshots are filtered to obtain a plurality of classified screenshot sets, including:

[0013] Obtaining the operation interface title corresponding to the operation interface screenshot;

[0014] Aggregate all the operation interface screenshots corresponding to the same operation interface title in the operation interface screenshot set to obtain an initial classification screenshot set;

[0015] The similar operation interface screenshots in the initial classification screenshot set are eliminated to obtain the classification screenshot set.

[0016] Optionally, the step of eliminating similar operation interface screenshots from the initial classified screenshot set to obtain the classified screenshot set includes:

[0017] Convert each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector;

[0018] Calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain the screenshot similarity;

[0019] Similar operation interface screenshots in the initial classification screenshot set are eliminated according to the screenshot similarity to obtain the corresponding classification screenshot set.

[0020] Optionally, the step of converting each operation interface screenshot in the initial classified screenshot set into a vector to obtain a screenshot vector includes:

[0021] Select any one of the operation interface screenshots in the initial classification screenshot set and input it into the pre-built deep learning network model;

[0022] The output values ​​of all nodes in the fully connected layer in the deep learning network model are obtained to construct a vector and obtain the corresponding screenshot vector.

[0023] Optionally, the similarity between any two operation interface screenshots in the initial classified screenshot set is calculated according to the screenshot vector to obtain the screenshot similarity, including:

[0024] The screenshot similarity is calculated using the following formula:

[0025]

[0026] Among them, X i Represents the i-th element of the screenshot vector X, Y i is the i-th element of the screenshot vector Y, Sim represents the similarity between the screenshot vector X corresponding to the operation interface screenshot X and the screenshot vector Y corresponding to the operation interface screenshot Y, and n represents the vector dimension of the screenshot vector X and the screenshot vector Y.

[0027] Optionally, before performing image region division and annotation on each operation interface screenshot in the classified screenshot set using a preset region division model, the method further comprises:

[0028] Acquire a historical operation interface screenshot set, wherein each historical operation interface screenshot in the historical operation interface screenshot set includes a division area coordinate and a division area annotation;

[0029] Selecting a historical operation interface screenshot from the historical operation interface screenshot set and inputting a preset initial region division model to obtain a division region prediction coordinate and a division region annotation prediction value;

[0030] Determine the true value of the divided area annotation according to the divided area annotation;

[0031] Calculating the annotation loss value using a preset first loss function according to the actual value of the divided area annotation and the predicted value of the divided area annotation;

[0032] Calculating the area division loss value by using a preset second loss function according to the division area coordinates and the prediction area coordinates;

[0033] A target loss value is obtained by calculating the annotation loss value and the region division loss value;

[0034] When the target loss value is greater than a preset loss threshold, the model parameters of the initial region division model are adjusted, and the process returns to the step of selecting a historical operation interface screenshot in the historical operation interface screenshot set to input a preset initial region division model;

[0035] When the target loss value is not greater than a preset loss threshold, the initial region division model is output to obtain the region division model.

[0036] Optionally, the step of combining and packaging all the annotated operation interface screenshots in the order of the corresponding screenshot times to obtain an operation guide file includes:

[0037] All the marked operation interface screenshots are stitched together in the order of the corresponding screenshot time to obtain an operation guide diagram;

[0038] The operation guide diagram is converted into a preset file format to obtain the operation guide file.

[0039] In order to solve the above problem, the present invention further provides a device for generating a device operation guide file, the device comprising:

[0040] The screenshot annotation module is used to obtain an operation interface screenshot set, wherein the operation interface screenshot set includes multiple operation interface screenshots, and each operation interface screenshot includes a corresponding screenshot time; the operation interface screenshots in the operation interface screenshot set are classified and similar screenshots are filtered to obtain multiple classified screenshot sets; each operation interface screenshot in the classified screenshot set is divided into image regions and annotated to obtain an annotated screenshot;

[0041] An image content description module is used to describe the image content of the annotated screenshot to obtain image content text; and use each image content text to annotate the corresponding operation interface screenshot;

[0042] The file generation module is used to combine and package all the annotated operation interface screenshots in the order of the corresponding screenshot time to obtain an operation guide file.

[0043] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0044] a memory storing at least one computer program; and

[0045] The processor executes the computer program stored in the memory to implement the above-mentioned method for generating the device operation guide file.

[0046] In order to solve the above problem, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned method for generating a device operation guide file.

[0047] The embodiment of the present invention performs operation interface classification and similar screenshot filtering on the operation interface screenshots in the operation interface screenshot set to obtain multiple classified screenshot sets, removes similar operation interface screenshots in the operation interface screenshot set, avoids repeated image content annotation, and improves the generation efficiency of device guidance files. Therefore, the device operation guidance file generation method, device, electronic device and readable storage medium proposed in the embodiment of the present invention improve the efficiency of device operation guidance file generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a flow chart of a method for generating a device operation guide file provided in an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a module of a device for generating a device operation guide file according to an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the internal structure of an electronic device for implementing a method for generating a device operation guide file provided by an embodiment of the present invention;

[0051] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0053] An embodiment of the present invention provides a method for generating a device operation guide file. The execution subject of the device operation guide file generation method includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the device operation guide file generation method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0054] Reference Figure 1 FIG. 1 is a flow chart of a method for generating a device operation guide file according to an embodiment of the present invention. In the embodiment of the present invention, the method for generating a device operation guide file includes:

[0055] S1. Acquire an operation interface screenshot set, wherein the operation interface screenshot set includes a plurality of operation interface screenshots, and each of the operation interface screenshots includes a corresponding screenshot time;

[0056] In detail, the operation interface screenshots in the embodiments of the present invention are screenshots of the test operation interface of certain smart devices when a certain software iteration version is subjected to UAT testing, wherein the smart devices include but are not limited to: bank branch devices, self-service cash register devices, etc. For example: bank branch device A, the iteration version is V1.0.1.0, when in the UAT testing phase, a screenshot of the operation interface is taken.

[0057] Specifically, in an embodiment of the present invention, during a UAT test phase of a certain iteration version of the software, a screenshot of an operation interface of the software can be taken according to the operation sequence of the software to obtain a screenshot set of the operation interface.

[0058] In another embodiment of the present invention, the operation interface screenshot set can be stored in a blockchain node, utilizing the high throughput characteristics of the blockchain node to improve data access efficiency.

[0059] S2, classifying the operation interface screenshots in the operation interface screenshot set and filtering similar screenshots to obtain multiple classified screenshot sets;

[0060] In the embodiment of the present invention, since the operation instructions corresponding to the operation screenshots corresponding to different operation interfaces are different, the operation screenshots of the same operation interface have the same operation interface title.

[0061] In detail, in the embodiment of the present invention, the operation interface title corresponding to the operation interface screenshot is obtained; all the operation interface screenshots corresponding to the same operation interface title in the operation interface screenshot set are aggregated to obtain an initial classification screenshot set.

[0062] Furthermore, since the operation interface screenshots of similar operation steps in the classified screenshot set are also similar, highly similar operation interface screenshots can be marked with the same operation instructions. Therefore, in order to improve the efficiency of generating operation instructions, it is necessary to eliminate similar operation interface screenshots in the initial classified screenshot set to obtain the classified screenshot set.

[0063] Furthermore, in the embodiment of the present invention, the classification screenshot set is obtained by eliminating similar operation interface screenshots in the initial classification screenshot set, including:

[0064] Convert each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector;

[0065] Calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain the screenshot similarity;

[0066] In detail, the embodiment of the present invention calculates the similarity of two screenshot vectors corresponding to the two operation interface screenshots to obtain corresponding screenshot similarities.

[0067] Optionally, the embodiment of the present invention uses the following formula to calculate the similarity:

[0068]

[0069] Among them, X i Represents the i-th element of the screenshot vector X, Y i is the i-th element of the screenshot vector Y, Sim represents the similarity between the screenshot vector X corresponding to the operation interface screenshot X and the screenshot vector Y corresponding to the operation interface screenshot Y, and n represents the vector dimension of the screenshot vector X and the screenshot vector Y.

[0070] Similar operation interface screenshots in the initial classification screenshot set are eliminated according to the screenshot similarity to obtain a corresponding classification screenshot set.

[0071] Furthermore, in an embodiment of the present invention, each operation interface screenshot in the initial classification screenshot set is converted into a vector to obtain a screenshot vector, including:

[0072] Select any one of the operation interface screenshots in the initial classification screenshot set and input it into the pre-built deep learning network model;

[0073] The output values ​​of all nodes in the fully connected layer in the deep learning network model are obtained to construct a vector and obtain the corresponding screenshot vector.

[0074] Optionally, the deep learning network model in the embodiment of the present invention is an artificial intelligence model, and the deep learning network model is CNN (Convolutional Neural Networks).

[0075] Furthermore, in an embodiment of the present invention, a pre-built deep learning network model is used to screen and filter similar operation interface screenshots in the initial classification screenshot set, including:

[0076] In detail, in the embodiment of the present invention, the output values ​​of all nodes in the fully connected layer in the deep learning network model are obtained to construct a vector to obtain the corresponding screenshot vector, including:

[0077] All the node output values ​​are combined vertically according to the order of the corresponding nodes in the fully connected layer to obtain the screenshot vector.

[0078] For example, the fully connected layer has 3 nodes, which are the first node, the second node, and the third node in order. The node output value set of the operation interface screenshot A has 3 node output values ​​of 3, 5, and 1, among which the node output value 1 is the output of the first node, the node output value 3 is the output of the second node, and the node output value 5 is the output of the third node. The three node output values ​​in the node output value set of the operation interface screenshot A are combined vertically in the node order to obtain the operation interface screenshot feature vector of the operation interface screenshot A.

[0079] In detail, in the embodiment of the present invention, similar operation interface screenshots in the initial classification screenshot set are eliminated according to the screenshot similarity to obtain a corresponding classification screenshot set, including:

[0080] Any one of the two operation interface screenshots corresponding to the screenshot similarity greater than a preset similarity threshold in the initial split screenshot set is eliminated to obtain the classified screenshot set. Optionally, in the embodiment of the present invention, the similarity threshold may be 90%. In the embodiment of the present invention, when the screenshot similarity corresponding to the two operation interface screenshots is greater than the similarity threshold, it means that the two operation interface screenshots are highly similar, and only one of them is retained.

[0081] S3, using a preset region division model to perform image region division and annotation on each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot;

[0082] The operation interface screenshot described in the embodiment of the present invention includes different page controls, such as: input box, single / double options, drop-down options, buttons, handwritten signature box, ID photo shooting page, etc. Due to different page controls and different operation instructions corresponding to different page controls, it is necessary to divide and annotate different page control areas in the operation interface screenshot. Therefore, the embodiment of the present invention divides and annotates the image area of ​​each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot.

[0083] In detail, in an embodiment of the present invention, a preset region division model is used to perform image region division and annotation on each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot. Optionally, the region division model is a Deep CNN model.

[0084] Furthermore, before the embodiment of the present invention uses a preset region division model to perform image region division and annotation on each operation interface screenshot in the classified screenshot set, the method further includes:

[0085] Acquire a historical operation interface screenshot set, wherein each historical operation interface screenshot in the historical operation interface screenshot set includes a division area coordinate and a division area annotation;

[0086] Optionally, the divided area annotations in the embodiment of the present invention include but are not limited to: input boxes, single / multiple options, drop-down options, buttons, handwritten signature boxes, ID photo shooting pages, etc.

[0087] Selecting a historical operation interface screenshot from the historical operation interface screenshot set and inputting a preset initial region division model to obtain a division region prediction coordinate and a division region annotation prediction value;

[0088] Determine the true value of the divided area annotation according to the divided area annotation;

[0089] For example: if the partition area annotation is "input box", then the corresponding partition area annotation true value is: input box: 1.

[0090] Calculating the annotation loss value using a preset first loss function according to the actual value of the divided area annotation and the predicted value of the divided area annotation;

[0091] Calculating the area division loss value by using a preset second loss function according to the division area coordinates and the prediction area coordinates;

[0092] Optionally, in the embodiment of the present invention, the first loss function and the second loss function may be: a logarithmic loss function, a square loss function, a cross entropy loss function, etc.

[0093] A target loss value is obtained by calculating the annotation loss value and the region division loss value;

[0094] When the target loss value is greater than a preset loss threshold, the model parameters of the initial region division model are adjusted, and the step of selecting the historical operation interface screenshot in the historical operation interface screenshot set to input the preset initial region division model is returned; when the target loss value is not greater than the preset loss threshold, the initial region division model is output to obtain the region division model.

[0095] S4, describing the image content of the annotated screenshot to obtain an image content text;

[0096] In detail, the embodiment of the present invention uses a preset image caption technology to describe the image content of the annotated screenshot to obtain the image content text. Optionally, the embodiment of the present invention uses GRU to describe the image content of the annotated screenshot, extracts each divided area in the annotated screenshot for intent recognition, obtains the intent text, combines the intent text with the annotation corresponding to the divided area, and obtains the divided area content text, such as: the annotation corresponding to the divided area is "text box", and the corresponding intent text is "fill in personal information", then the corresponding divided area content text is "fill in personal information in the text box", and further, all the divided area content texts corresponding to the annotated screenshot are combined to obtain the image content text.

[0097] For example: the annotation screenshot contains a name input box and a product type radio button, then an image description is performed on the knowledge screenshot, and the resulting image content text is to enter a name in the input box and select a product type that needs to be processed.

[0098] S5, using each of the image content texts to annotate the corresponding operation interface screenshot;

[0099] In detail, in order to associate the image content text with the corresponding operation interface screenshot, the embodiment of the present invention facilitates the user to browse the image and text by comparison, and uses each image content text to annotate the corresponding operation interface screenshot. In detail, the embodiment of the present invention constructs an annotation text box in the operation interface screenshot, and inputs the image content text into the annotation text box to annotate the operation interface screenshot.

[0100] S6. Combining and packaging all the marked operation interface screenshots according to the chronological order of the corresponding screenshot times to obtain an operation guide file.

[0101] In detail, in the embodiment of the present invention, all the annotated operation interface screenshots are stitched in the order of the corresponding screenshot time to obtain an operation guide map; the operation guide map is converted into a preset file format to obtain the operation guide file. Optionally, the preset file format in the embodiment of the present invention can be PDF or doc format.

[0102] like Figure 2 The figure shows a functional module diagram of the device operation guide file generating apparatus of the present invention.

[0103] The device operation guide file generation device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the device operation guide file generation device can include a screenshot annotation module 101, an image content description module 102, and a file generation module 103. The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0104] In this embodiment, the functions of each module / unit are as follows:

[0105] The screenshot annotation module 101 is used to obtain an operation interface screenshot set, wherein the operation interface screenshot set includes multiple operation interface screenshots, and each operation interface screenshot includes a corresponding screenshot time; the operation interface screenshots in the operation interface screenshot set are classified and similar screenshots are filtered to obtain multiple classified screenshot sets; each operation interface screenshot in the classified screenshot set is divided into image regions and annotated to obtain an annotated screenshot;

[0106] In detail, the operation interface screenshots in the embodiments of the present invention are screenshots of the test operation interface of certain smart devices when a certain software iteration version is subjected to UAT testing, wherein the smart devices include but are not limited to: bank branch devices, self-service cash register devices, etc. For example: bank branch device A, the iteration version is V1.0.1.0, when in the UAT testing phase, a screenshot of the operation interface is taken.

[0107] Specifically, in an embodiment of the present invention, during a UAT test phase of a certain iteration version of the software, a screenshot of an operation interface of the software can be taken according to the operation sequence of the software to obtain a screenshot set of the operation interface.

[0108] In another embodiment of the present invention, the operation interface screenshot set can be stored in a blockchain node, utilizing the high throughput characteristics of the blockchain node to improve data access efficiency.

[0109] In the embodiment of the present invention, since the operation instructions corresponding to the operation screenshots corresponding to different operation interfaces are different, the operation screenshots of the same operation interface have the same operation interface title.

[0110] In detail, the screenshot annotation module 101 in the embodiment of the present invention obtains the operation interface title corresponding to the operation interface screenshot; and aggregates all the operation interface screenshots corresponding to the same operation interface title in the operation interface screenshot set to obtain an initial classification screenshot set.

[0111] Furthermore, since the operation interface screenshots of similar operation steps in the classified screenshot set are also similar, highly similar operation interface screenshots can be marked with the same operation instructions. Therefore, in order to improve the efficiency of generating operation instructions, the screenshot annotation module 101 needs to eliminate similar operation interface screenshots in the initial classified screenshot set to obtain the classified screenshot set.

[0112] Furthermore, the screenshot annotation module 101 in the embodiment of the present invention eliminates similar operation interface screenshots in the initial classified screenshot set to obtain the classified screenshot set, including:

[0113] Convert each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector;

[0114] Calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain the screenshot similarity;

[0115] In detail, the embodiment of the present invention calculates the similarity of two screenshot vectors corresponding to the two operation interface screenshots to obtain corresponding screenshot similarities.

[0116] Optionally, the embodiment of the present invention uses the following formula to calculate the similarity:

[0117]

[0118] Among them, X i Represents the i-th element of the screenshot vector X, Y i is the i-th element of the screenshot vector Y, Sim represents the similarity between the screenshot vector X corresponding to the operation interface screenshot X and the screenshot vector Y corresponding to the operation interface screenshot Y, and n represents the vector dimension of the screenshot vector X and the screenshot vector Y.

[0119] Similar operation interface screenshots in the initial classification screenshot set are eliminated according to the screenshot similarity to obtain a corresponding classification screenshot set.

[0120] Furthermore, the screenshot annotation module 101 in the embodiment of the present invention converts each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector, including:

[0121] Select any one of the operation interface screenshots in the initial classification screenshot set and input it into the pre-built deep learning network model;

[0122] The output values ​​of all nodes in the fully connected layer in the deep learning network model are obtained to construct a vector and obtain the corresponding screenshot vector.

[0123] Optionally, the deep learning network model in the embodiment of the present invention is an artificial intelligence model, and the deep learning network model is CNN (Convolutional Neural Networks).

[0124] Furthermore, in the embodiment of the present invention, the screenshot annotation module 101 uses a pre-built deep learning network model to screen and filter similar operation interface screenshots in the initial classification screenshot set, including:

[0125] In detail, in the embodiment of the present invention, the output values ​​of all nodes in the fully connected layer in the deep learning network model are obtained to construct a vector to obtain the corresponding screenshot vector, including:

[0126] All the node output values ​​are combined vertically according to the order of the corresponding nodes in the fully connected layer to obtain the screenshot vector.

[0127] For example, the fully connected layer has 3 nodes, which are the first node, the second node, and the third node in order. The node output value set of the operation interface screenshot A has 3 node output values ​​of 3, 5, and 1, among which the node output value 1 is the output of the first node, the node output value 3 is the output of the second node, and the node output value 5 is the output of the third node. The three node output values ​​in the node output value set of the operation interface screenshot A are combined vertically in the node order to obtain the operation interface screenshot feature vector of the operation interface screenshot A.

[0128] In detail, the screenshot annotation module 101 in the embodiment of the present invention removes similar operation interface screenshots from the initial classified screenshot set according to the screenshot similarity to obtain a corresponding classified screenshot set, including:

[0129] Any one of the two operation interface screenshots corresponding to the screenshot similarity greater than a preset similarity threshold in the initial split screenshot set is eliminated to obtain the classified screenshot set. Optionally, in the embodiment of the present invention, the similarity threshold may be 90%. In the embodiment of the present invention, when the screenshot similarity corresponding to the two operation interface screenshots is greater than the similarity threshold, it means that the two operation interface screenshots are highly similar, and only one of them is retained.

[0130] The operation interface screenshot described in the embodiment of the present invention includes different page controls, such as: input box, single / double options, drop-down options, buttons, handwritten signature box, ID photo shooting page, etc. Due to different page controls and different operation instructions corresponding to different page controls, it is necessary to divide and annotate different page control areas in the operation interface screenshot. Therefore, the embodiment of the present invention divides and annotates the image area of ​​each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot.

[0131] In detail, the screenshot annotation module 101 in the embodiment of the present invention uses a preset region division model to perform image region division and annotation on each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot. Optionally, the region division model is a Deep CNN model.

[0132] Furthermore, before the screenshot annotation module 101 in the embodiment of the present invention uses a preset region division model to perform image region division and annotation on each operation interface screenshot in the classified screenshot set, the method further includes:

[0133] Acquire a historical operation interface screenshot set, wherein each historical operation interface screenshot in the historical operation interface screenshot set includes a division area coordinate and a division area annotation;

[0134] Optionally, the divided area annotations in the embodiment of the present invention include but are not limited to: input boxes, single / multiple options, drop-down options, buttons, handwritten signature boxes, ID photo shooting pages, etc.

[0135] Selecting a historical operation interface screenshot from the historical operation interface screenshot set and inputting a preset initial region division model to obtain a division region prediction coordinate and a division region annotation prediction value;

[0136] Determine the true value of the divided area annotation according to the divided area annotation;

[0137] For example: if the partition area annotation is "input box", then the corresponding partition area annotation true value is: input box: 1.

[0138] Calculating the annotation loss value using a preset first loss function according to the actual value of the divided area annotation and the predicted value of the divided area annotation;

[0139] Calculating the area division loss value by using a preset second loss function according to the division area coordinates and the prediction area coordinates;

[0140] Optionally, in the embodiment of the present invention, the first loss function and the second loss function may be: a logarithmic loss function, a square loss function, a cross entropy loss function, etc.

[0141] A target loss value is obtained by calculating the annotation loss value and the region division loss value;

[0142] When the target loss value is greater than a preset loss threshold, the model parameters of the initial region division model are adjusted, and the step of selecting the historical operation interface screenshot in the historical operation interface screenshot set to input the preset initial region division model is returned; when the target loss value is not greater than the preset loss threshold, the initial region division model is output to obtain the region division model.

[0143] The image content description module 102 is used to describe the image content of the annotated screenshot to obtain image content text; and use each image content text to annotate the corresponding operation interface screenshot;

[0144] In detail, the image content description module 102 in the embodiment of the present invention uses a preset image description (ImageCaption) technology to describe the image content of the annotation screenshot to obtain the image content text. Optionally, the embodiment of the present invention uses GRU to describe the image content of the annotation screenshot, extracts each divided area in the annotation screenshot for intent recognition, obtains the intent text, combines the intent text with the annotation corresponding to the divided area, and obtains the divided area content text, such as: the annotation corresponding to the divided area is "text box", and the corresponding intent text is "fill in personal information", then the corresponding divided area content text is "fill in personal information in the text box", and further, all the divided area content texts corresponding to the annotation screenshot are combined to obtain the image content text.

[0145] For example: the annotation screenshot contains a name input box and a product type radio button, then an image description is performed on the knowledge screenshot, and the resulting image content text is to enter a name in the input box and select a product type that needs to be processed.

[0146] In detail, in order to associate the image content text with the corresponding operation interface screenshot and facilitate the user to browse the image and text, the image content description module 102 uses each image content text to annotate the corresponding operation interface screenshot. In detail, the embodiment of the present invention constructs an annotation text box in the operation interface screenshot, and inputs the image content text into the annotation text box to annotate the operation interface screenshot.

[0147] The file generation module 103 is used to combine and package all the annotated operation interface screenshots according to the chronological order of the corresponding screenshot times to obtain an operation guide file.

[0148] In detail, the file generation module 103 in the embodiment of the present invention stitches all the annotated operation interface screenshots in the order of the corresponding screenshot time to obtain an operation guide map; and converts the operation guide map into a preset file format to obtain the operation guide file. Optionally, the preset file format in the embodiment of the present invention can be PDF or doc format.

[0149] like Figure 2 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing the method for generating a device operation guide file according to the present invention.

[0150] The electronic device may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a device operation guide file generation program.

[0151] Wherein, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 11 may be an internal storage unit of an electronic device in some embodiments, such as a mobile hard disk of the electronic device. The memory 11 may also be an external storage device of an electronic device in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device, such as the code of the device operation guide file generation program, but also be used to temporarily store data that has been output or is to be output.

[0152] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules (such as device operation guide file generation programs, etc.) stored in the memory 11, and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0153] The communication bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The communication bus 12 is configured to realize connection and communication between the memory 11 and at least one processor 10, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0154] Figure 2 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 2 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0155] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure classification circuits, power converters or inverters, power status indicators, etc. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0156] Optionally, the communication interface 13 may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.

[0157] Optionally, the communication interface 13 may also include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0158] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0159] The device operation guide file generation program stored in the memory 11 in the electronic device is a combination of multiple computer programs. When running in the processor 10, it can achieve:

[0160] Acquire an operation interface screenshot set, wherein the operation interface screenshot set includes multiple operation interface screenshots, and each operation interface screenshot includes a corresponding screenshot time;

[0161] Classifying the operation interface screenshots in the operation interface screenshot set and filtering similar screenshots to obtain a plurality of classified screenshot sets;

[0162] Performing image region division and annotation on each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot;

[0163] Describing the image content of the annotated screenshot to obtain image content text;

[0164] Annotate the corresponding operation interface screenshot with the text of each image content;

[0165] All the annotated operation interface screenshots are combined and packaged in the order of the corresponding screenshot time to obtain an operation guide file.

[0166] Specifically, the specific implementation method of the processor 10 for the above computer program can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0167] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0168] An embodiment of the present invention may further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program may implement:

[0169] Acquire an operation interface screenshot set, wherein the operation interface screenshot set includes multiple operation interface screenshots, and each operation interface screenshot includes a corresponding screenshot time;

[0170] Classifying the operation interface screenshots in the operation interface screenshot set and filtering similar screenshots to obtain a plurality of classified screenshot sets;

[0171] Performing image region division and annotation on each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot;

[0172] Describing the image content of the annotated screenshot to obtain image content text;

[0173] Annotate the corresponding operation interface screenshot with the text of each image content;

[0174] All the annotated operation interface screenshots are combined and packaged in the order of the corresponding screenshot time to obtain an operation guide file.

[0175] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0176] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0177] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0179] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0180] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0181] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0182] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0183] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for generating a device operation guide file, characterized in that: The method comprises: Acquire an operation interface screenshot set, wherein the operation interface screenshot set includes multiple operation interface screenshots, and each operation interface screenshot includes a corresponding screenshot time; Using a pre-built deep learning network model, the operation interface screenshots in the operation interface screenshot set are classified and similar screenshots are filtered to obtain a plurality of classified screenshot sets; Performing image region division and annotation on each operation interface screenshot in the classified screenshot set by using a preset region division model to obtain an annotated screenshot; Using a preset picture description model to describe the image content of the annotation screenshot to obtain image content text; Annotate the corresponding operation interface screenshot with the text of each image content; All the annotated operation interface screenshots are combined and packaged in the order of the corresponding screenshot time to obtain an operation guide file.

2. The method for generating a device operation guide file according to claim 1, characterized in that: The step of classifying the operation interface screenshots in the operation interface screenshot set and filtering similar screenshots to obtain a plurality of classified screenshot sets includes: Obtaining the operation interface title corresponding to the operation interface screenshot; Aggregate all the operation interface screenshots corresponding to the same operation interface title in the operation interface screenshot set to obtain an initial classification screenshot set; The similar operation interface screenshots in the initial classification screenshot set are eliminated to obtain the classification screenshot set.

3. The method for generating a device operation guide file according to claim 2, characterized in that: The method of eliminating similar operation interface screenshots from the initial classification screenshot set to obtain the classification screenshot set includes: Convert each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector; Calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain the screenshot similarity; Similar operation interface screenshots in the initial classification screenshot set are eliminated according to the screenshot similarity to obtain the corresponding classification screenshot set.

4. The method for generating a device operation guide file according to claim 2, wherein: The method of converting each operation interface screenshot in the initial classification screenshot set into a vector to obtain a screenshot vector includes: Select any one of the operation interface screenshots in the initial classification screenshot set and input it into the pre-built deep learning network model; The output values ​​of all nodes in the fully connected layer in the deep learning network model are obtained to construct a vector and obtain the corresponding screenshot vector.

5. The method for generating a device operation guide file according to claim 3, characterized in that: The calculating the similarity of any two operation interface screenshots in the initial classification screenshot set according to the screenshot vector to obtain the screenshot similarity includes: The screenshot similarity is calculated using the following formula: Among them, X i Represents the i-th element of the screenshot vector X, Y i is the i-th element of the screenshot vector Y, Sim represents the similarity between the screenshot vector X corresponding to the operation interface screenshot X and the screenshot vector Y corresponding to the operation interface screenshot Y, and n represents the vector dimension of the screenshot vector X and the screenshot vector Y.

6. The method for generating a device operation guide file according to claim 1, characterized in that: Before performing image region division and annotation on each operation interface screenshot in the classified screenshot set using a preset region division model, the method further includes: Acquire a historical operation interface screenshot set, wherein each historical operation interface screenshot in the historical operation interface screenshot set includes a division area coordinate and a division area annotation; Selecting a historical operation interface screenshot from the historical operation interface screenshot set and inputting a preset initial region division model to obtain a division region prediction coordinate and a division region annotation prediction value; Determine the true value of the divided area annotation according to the divided area annotation; Calculating the annotation loss value using a preset first loss function according to the actual value of the divided area annotation and the predicted value of the divided area annotation; Calculating the area division loss value by using a preset second loss function according to the division area coordinates and the division area prediction coordinates; A target loss value is obtained by calculating the annotation loss value and the region division loss value; When the target loss value is greater than a preset loss threshold, the model parameters of the initial region division model are adjusted, and the process returns to the step of selecting a historical operation interface screenshot in the historical operation interface screenshot set to input a preset initial region division model; When the target loss value is not greater than a preset loss threshold, the initial region division model is output to obtain the region division model.

7. The method for generating a device operation guide file according to any one of claims 1 to 6, characterized in that: The operation interface screenshots after all the annotations are combined and packaged in the order of the corresponding screenshot time to obtain an operation guide file, including: All the marked operation interface screenshots are stitched together in the order of the corresponding screenshot time to obtain an operation guide diagram; The operation guide diagram is converted into a preset file format to obtain the operation guide file.

8. A device for generating a device operation guide file, characterized in that: include: A screenshot annotation module is used to obtain an operation interface screenshot set, wherein the operation interface screenshot set includes multiple operation interface screenshots, and each operation interface screenshot includes a corresponding screenshot time; using a pre-built deep learning network model to classify the operation interface screenshots in the operation interface screenshot set and filter similar screenshots to obtain multiple classified screenshot sets; using a preset area division model to perform image area division and annotation on each operation interface screenshot in the classified screenshot set to obtain an annotated screenshot; An image content description module is used to describe the image content of the annotation screenshot using a preset picture description model to obtain image content text; and to annotate the corresponding operation interface screenshot using each image content text; The file generation module is used to combine and package all the annotated operation interface screenshots in the order of the corresponding screenshot time to obtain an operation guide file.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the device operation guide file generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a device operation guide file according to any one of claims 1 to 7 is implemented.

Images