Method, device, equipment and medium for generating operation instructions based on image recognition

Abstract

The present invention relates to artificial intelligence technology, and discloses a method for generating operation instructions based on image recognition, including: obtaining a page screenshot of each process node in a product purchase process, and identifying the image text in the page screenshot; segmenting the image text into text segmentation; obtaining the operation instruction text of each process node in the product purchase process, and extracting the text semantics of the operation instruction text; using the text semantics to construct a decision tree model, and using the decision tree model to filter the text segmentation to obtain text keywords; using the operation instruction text to mark the position of the text keyword in the page screenshot, and collecting all the marked page screenshots as the interface operation instructions of the product purchase process. In addition, the present invention also relates to blockchain technology, and the page screenshot can be stored in the node of the blockchain. The present invention also proposes an operation instruction generation device, an electronic device and a medium based on image recognition. The present invention can improve the accuracy of operation instructions for users.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and computer-readable storage medium for generating operation instructions based on image recognition. Background Art

[0002] With the diversification of user needs, more and more products have emerged in the market. However, for many products, there is a relatively complicated purchase process. For example, financial products sold by banks or securities companies, or products that require multiple models and styles to be selected when purchasing, resulting in a complicated purchase process, etc. Therefore, how to provide users with operational guidance to avoid operational errors due to complicated processes has become a focus of attention.

[0003] In the existing methods, most of them generate operating instructions according to the operating process and explain them to users in the form of operating instructions. However, in this method, the generated operating instructions are often complicated and users cannot accurately know the operating process after reading them. Therefore, how to improve the accuracy of operating guidance for users has become a problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a method and device for generating an operation guide based on image recognition, and a computer-readable storage medium, the main purpose of which is to solve the problem of low accuracy when providing operation guides to users.

[0005] To achieve the above object, the present invention provides a method for generating operation instructions based on image recognition, comprising:

[0006] Obtaining a page screenshot of each process node in the product purchase process, and identifying image text in the page screenshot;

[0007] Performing word segmentation on the image text to obtain text word segmentation;

[0008] Obtaining the operation guide text of each process node in the product purchase process, and extracting the text semantics of the operation guide text;

[0009] Using the text semantics to construct a decision tree model, and using the decision tree model to screen the text segmentation to obtain text keywords;

[0010] The operation guide text is used to mark the position of the text keyword in the page screenshot, and all the marked page screenshots are collected as the interface operation guide of the product purchase process.

[0011] Optionally, the identifying the image text in the page screenshot includes:

[0012] Dividing the page screenshot into image blocks to obtain a plurality of image blocks;

[0013] Performing convolution and pooling processing on each of the multiple image blocks respectively to obtain image features;

[0014] Calculating the distance value between the image feature and a preset text feature, and collecting image blocks whose distance value is less than a preset distance threshold as text image blocks;

[0015] Binarizing the text image block, and dividing the binarized text image block into text blocks to obtain a single-word image;

[0016] The single-word image is subjected to text recognition using a preset text recognition model, and the text recognition results are collected as graphic text.

[0017] Optionally, the binarization processing of the text image block includes:

[0018] Selecting one pixel point from the text image block one by one as a target pixel point;

[0019] Determine whether the pixel value of the target pixel is greater than a preset pixel threshold;

[0020] If the pixel value of the target pixel point is greater than the pixel threshold, the pixel value of the target pixel point is set to a first parameter value;

[0021] If the pixel value of the target pixel point is less than or equal to the pixel threshold, the pixel value of the target pixel point is set to a second parameter value.

[0022] Optionally, the segmenting the image text to obtain text segmentation includes:

[0023] Segmenting the image text according to different preset lengths to obtain segmented texts;

[0024] The segmented text is searched in a preset dictionary, and the segmented text retrieved in the preset dictionary is used as the text segmentation of the image text.

[0025] Optionally, extracting text semantics of the operation guide text includes:

[0026] Segmenting the operation guide text to obtain operation guide segmentation words;

[0027] Convert each word in the operation guide word segmentation into a word segmentation vector, and perform length unification processing on the word segmentation vector to obtain a uniform length vector;

[0028] Each of the uniform-length vectors is used as a row vector to splice into a vector matrix, and the vector matrix is ​​used as the text semantics of the operation guidance text.

[0029] Optionally, performing length unification processing on the word segmentation vector to obtain a uniform length vector includes:

[0030] Counting the vector length of each of the word segmentation vectors, and determining the word segmentation vector with the largest vector length as the target vector;

[0031] The vector length of each vector in the word segmentation vector is extended to the vector length of the target vector using preset parameters to obtain a uniform length vector.

[0032] Optionally, constructing a decision tree model using the text semantics includes:

[0033] Select one of the process nodes from the product purchase process one by one as the target node;

[0034] Assigning a value to a preset decision function using the text semantics of the operation guidance text of the target node, and using the assigned decision function as a decision condition to generate a decision tree for the target node;

[0035] The decision tree that brings together all process nodes in the product purchase process is a decision tree model.

[0036] In order to solve the above problems, the present invention further provides an operation guide generating device based on image recognition, the device comprising:

[0037] A text recognition module is used to obtain a page screenshot of each process node in the product purchase process and recognize the image text in the page screenshot;

[0038] A text segmentation module, used for segmenting the image text to obtain text segmentation;

[0039] A semantic extraction module, used to obtain the operation guide text of each process node in the product purchase process, and extract the text semantics of the operation guide text;

[0040] A text screening module, used to construct a decision tree model using the text semantics, and use the decision tree model to screen the text segmentation to obtain text keywords;

[0041] The guidance marking module is used to mark the position of the text keywords in the page screenshot using the operation guidance text, and to collect all the marked page screenshots into the interface operation guidance of the product purchase process.

[0042] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0043] at least one processor; and,

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned method for generating operation instructions based on image recognition.

[0046] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned method for generating operation instructions based on image recognition.

[0047] The embodiment of the present invention can perform image recognition on the page screenshots of different process nodes in the product purchase process respectively, and use the obtained operation guide text of each process node to build a decision tree model, so as to use the decision tree model to filter the text obtained by image recognition to obtain the keywords that need to be marked, and then use the operation guide text to mark the keywords in the page screenshot of each process node, so as to achieve the accurate marking of the content in the process node by using the operation guide text, and improve the accuracy of the operation guide. Therefore, the operation guide generation method, device, electronic device and computer-readable storage medium based on image recognition proposed by the present invention can solve the problem of low accuracy when providing operation guidance to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a flow chart of a method for generating an operation guide based on image recognition provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a process for binarizing a text image block provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a process for extracting text semantics provided by an embodiment of the present invention;

[0051] Figure 4 A functional module diagram of an operation guide generating device based on image recognition provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of the structure of an electronic device for implementing the method for generating operation guidance based on image recognition provided by an embodiment of the present invention.

[0053] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0055] The embodiment of the present application provides a method for generating an operation guide based on image recognition. The execution subject of the method for generating an operation guide based on image recognition includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for generating an operation guide based on image recognition can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0056] Reference Figure 1 FIG. 1 is a flow chart of a method for generating an operation guide based on image recognition according to an embodiment of the present invention. In this embodiment, the method for generating an operation guide based on image recognition includes:

[0057] S1. Obtain a page screenshot of each process node in the product purchase process, and identify the image text in the page screenshot.

[0058] In the embodiment of the present invention, the interface screenshots may be screenshots of pages (such as mobile app pages, computer website pages, etc.) at different process nodes in the purchase process of any product. For example, when a user is purchasing product A, a screenshot of the purchase interface of product A, a screenshot of the type selection interface of product A, a screenshot of the payment interface of product A, etc.

[0059] In detail, a computer statement with a data capture function (such as a Java statement, a Python statement, etc.) can be used to capture pre-stored page screenshots from a pre-built storage area, and the storage area includes but is not limited to data, blockchain nodes, and network caches.

[0060] Specifically, the page screenshot may be a pre-acquired unannotated page screenshot of each process node in the product purchase process, and the embodiment of the present invention may subsequently generate corresponding annotations based on the screenshot, so that when the user performs the actual purchase operation, the product can be purchased according to the pre-annotated content.

[0061] In an embodiment of the present invention, the page screenshot may be subjected to image recognition by using OCR (Optical Character Recognition) technology, deep learning technology based on an attention mechanism, or the like, so as to obtain the image text contained in the page screenshot.

[0062] In one embodiment of the present invention, the identifying the image text in the page screenshot includes:

[0063] Dividing the page screenshot into image blocks to obtain a plurality of image blocks;

[0064] Performing convolution and pooling processing on each of the multiple image blocks respectively to obtain image features;

[0065] Calculating the distance value between the image feature and a preset text feature, and collecting image blocks whose distance value is less than a preset distance threshold as text image blocks;

[0066] Binarizing the text image block, and dividing the binarized text image block into text blocks to obtain a single-word image;

[0067] The single-word image is subjected to text recognition using a preset text recognition model, and the text recognition results are collected as graphic text.

[0068] In detail, since not every area in the page screenshot contains text, directly performing text recognition on all the contents in the page screenshot will take up a large amount of computing resources and result in low recognition efficiency. Therefore, the page screenshot can be segmented into images according to a preset size to obtain multiple image blocks for each page screenshot.

[0069] Furthermore, convolution, pooling and other operations can be performed on each of the multiple image blocks to extract image features in each image block, and a preset distance value algorithm can be used to calculate the distance value between the image feature and the preset text feature to determine whether the image block contains text based on the distance value.

[0070] In the embodiment of the present invention, the calculating of the distance value between the image feature and the preset text feature includes:

[0071] The distance value between the image feature and the preset text feature is calculated using the following distance algorithm:

[0072]

[0073] Wherein, D is the distance value, a is the image feature, and b is the preset text feature.

[0074] In detail, the image block whose distance value is less than the preset distance threshold can be selected as the image block containing text, and the selected text image block can be binarized to remove the background information contained in the text image block, thereby improving the accuracy of recognizing the text in the text image block.

[0075] In the embodiment of the present invention, Figure 2 As shown, the binarization process of the text image block includes:

[0076] S21, selecting one pixel point from the text image block one by one as a target pixel point;

[0077] S22, determining whether the pixel value of the target pixel is greater than a preset pixel threshold;

[0078] If the pixel value of the target pixel point is greater than the pixel threshold, executing S23, setting the pixel value of the target pixel point to a first parameter value;

[0079] If the pixel value of the target pixel point is less than or equal to the pixel threshold, executing S24, setting the pixel value of the target pixel point to a second parameter value.

[0080] In detail, after the text image block is binarized, the shape of the text in the text image block can be highlighted, but the obtained binary image may contain multiple characters, and since there are many complex characters in Chinese characters, in order to improve the recognition efficiency of the text image, the binarized text image can be segmented according to single characters to obtain single-character images.

[0081] In an embodiment of the present invention, a preset text recognition model can be used to perform text recognition on the single-word image, and the results of the text recognition are compiled into a graphic text. The text recognition model refers to but is not limited to an artificial intelligence model with a glyph recognition function, and the text recognition model includes but is not limited to a CTC (Connectionist temporal classification) model and a Text-RCNN model.

[0082] S2. Segment the image text to obtain text segmentation.

[0083] In an embodiment of the present invention, since the text identified from the page screenshot may contain a large amount of text data, when generating page operation instructions, it is only necessary to mark the key text data in the page to improve the accuracy of the instructions. Therefore, the image text can be segmented to obtain text segmentation of the image text, so as to facilitate the subsequent screening of important keywords from the text segmentation to generate the page operation instructions.

[0084] In the embodiment of the present invention, the segmenting of the image text to obtain text segmentation includes:

[0085] Segmenting the image text according to different preset lengths to obtain segmented texts;

[0086] The segmented text is searched in a preset dictionary, and the segmented text retrieved in the preset dictionary is used as the text segmentation of the image text.

[0087] In detail, since there may be longer sentences in the image text, the image text can be segmented into multiple shorter segmented texts according to a preset length, and then the segmented texts can be searched in a preset dictionary. If the segmented texts can be retrieved, the retrieved segmented texts can be used as text segmentation of the image text, wherein the preset dictionary is a pre-acquired dictionary containing multiple standard entries.

[0088] For example, there is an image text: "Click to buy". The image text can be segmented according to length 2 to obtain segmented texts "click" and "buy". The image text can be segmented again according to length 3 to obtain segmented texts "click to buy" and "click to buy". The segmented texts are searched in the preset dictionary respectively. The segmented texts "click" and "buy" can be retrieved. Then the segmented texts "click" and "buy" that can be retrieved are used as text segmentations of the image text.

[0089] S3. Obtain the operation guide text of each process node in the product purchase process, and extract the text semantics of the operation guide text.

[0090] In the embodiment of the present invention, the operation guidance text is a pre-acquired text having a guidance effect on the operation of each process node in the product purchase process.

[0091] For example, for a purchase button on a page, you can get the operation guidance text "Click here to purchase" or "Click here to jump to the payment page"; for a button to select a product type on a page, you can get the operation guidance text "Click here to filter product types" or "Click here to select a product".

[0092] In detail, the step of obtaining the operation guide text of each process node in the product purchase process is consistent with the step of obtaining the page screenshot of each process node in the product purchase process in S1, and will not be repeated here.

[0093] In the embodiment of the present invention, in order to improve the accuracy of the generated interface operation guide, the text semantics of the obtained operation guide text may be extracted, and then the text segmentation may be screened according to the text semantics.

[0094] In the embodiment of the present invention, Figure 3 As shown, the extracting text semantics of the operation guide text includes:

[0095] S31, segmenting the operation guide text to obtain operation guide segmentation;

[0096] S32, converting each word in the operation guide word segmentation into a word segmentation vector, and performing length unification processing on the word segmentation vector to obtain a uniform length vector;

[0097] S33. Each of the uniform length vectors is used as a row vector to splice into a vector matrix, and the vector matrix is ​​used as the text semantics of the operation guidance text.

[0098] In detail, the step of segmenting the operation guide text to obtain operation guide segmentation is consistent with the step of segmenting the image text to obtain text segmentation in S2, and will not be repeated here.

[0099] Specifically, each word in the operation guide word segmentation can be converted into a word segmentation vector using a preset word2vec algorithm, bert algorithm, or one-hot algorithm.

[0100] In an embodiment of the present invention, since the operation guidance word segmentation contains multiple word segmentations, and the length of the word segmentation vector obtained by each word segmentation conversion is not necessarily the same, in order to facilitate the subsequent analysis of the word segmentations, preset parameters can be used to extend the vector length of each word segmentation vector in the word segmentation vector to a uniform length.

[0101] In the embodiment of the present invention, the step of performing length unification processing on the word segmentation vector to obtain a uniform length vector includes:

[0102] Counting the vector length of each of the word segmentation vectors, and determining the word segmentation vector with the largest vector length as the target vector;

[0103] The vector length of each vector in the word segmentation vector is extended to the vector length of the target vector using preset parameters to obtain a uniform length vector.

[0104] For example, the word segmentation vector includes word segmentation vector A: (1, 2), word segmentation vector B: (2, 5, 6, 8), and word segmentation vector C: (6, 3, 9); wherein the vector length of word segmentation vector A is 2, the vector length of word segmentation vector B is 4, and the length of word segmentation vector C is 3, then word segmentation vector B can be selected as the target vector, and when the preset parameter is 0, the preset parameter can be used to extend the word segmentation vector A to (1, 2, 0, 0), and the preset parameter can be used to extend the word segmentation vector C to (6, 3, 9, 0).

[0105] Furthermore, the extended word segmentation vector A, word segmentation vector B, and word segmentation vector C can be used as row vectors and concatenated into the following vector matrix:

[0106]

[0107] In the embodiment of the present invention, the vector matrix obtained by concatenation can be used as the text semantics of the operation guidance text.

[0108] S4. Utilize the text semantics to construct a decision tree model, and utilize the decision tree model to screen the text segmentation words to obtain text keywords.

[0109] In an embodiment of the present invention, in order to accurately mark the operation guidance process in the page screenshot, a decision tree model can be constructed using the text semantics, and then the decision tree model can be used to filter the text segmentation to obtain text keywords that need to be marked for guidance.

[0110] In the embodiment of the present invention, the step of constructing a decision tree model using the text semantics includes:

[0111] Select one of the process nodes from the product purchase process one by one as the target node;

[0112] Assigning a value to a preset decision function using the text semantics of the operation guidance text of the target node, and using the assigned decision function as a decision condition to generate a decision tree for the target node;

[0113] The decision tree that brings together all process nodes in the product purchase process is a decision tree model.

[0114] Exemplarily, the decision function may be:

[0115]

[0116] Among them, f(x) is the output value of the decision function, x is the parameter of the decision function, and g(y) is the input value of the decision function.

[0117] Specifically, one of the process nodes in the product purchase process can be selected one by one as the target node, and the text semantics corresponding to the operation guidance text of the target node is used to assign a value to the parameter x of the decision function, and the assigned decision function is used as a decision condition to generate the following decision tree:

[0118] When the input value g(y) of the decision tree matches the parameter x of the decision tree, the output value of the decision tree is f(x)=α;

[0119] When the input g(y) of the decision tree does not match the parameter x of the decision tree, the decision tree outputs a value f(x)=β.

[0120] In the embodiment of the present invention, the decision trees corresponding to each of the product features may be aggregated in parallel to obtain a decision tree model.

[0121] Furthermore, the decision tree model may be used to screen the text segmentation words to obtain text keywords.

[0122] For example, the decision tree model includes decision trees for process node A and process node B, wherein the decision tree corresponding to process node A is decision tree a, and the decision tree corresponding to process node B is decision tree b, and decision tree a is selected as the target decision tree; the input value (text segmentation) is input into the decision tree a, and an output result (text keyword) in which the input value output by the decision tree a is the same as the parameter of the decision tree a is obtained; the input value (text segmentation) is input into the decision tree b, and an output result (text keyword) in which the input value output by the decision tree b is different from the parameter of the decision tree b is obtained, and then the text keyword output by each process node is used as the keyword that the process node needs to be marked with the operation guidance text.

[0123] S5. Use the operation guide text to mark the position of the text keyword in the page screenshot, and collect all the marked page screenshots into the interface operation guide of the product purchase process.

[0124] In an embodiment of the present invention, the operation guide text can be used to mark the text keywords at the position of the text keywords in the page screenshot in the form of a text box or a label, so that when the user browses the page screenshot, the operation guide of each page can be quickly found.

[0125] Furthermore, the page screenshots after each process node in the product purchase process is marked can be collected to obtain the interface operation guidance of the product purchase process.

[0126] The embodiment of the present invention can perform image recognition on the page screenshots of different process nodes in the product purchase process respectively, and use the obtained operation guide text of each process node to build a decision tree model, so as to use the decision tree model to filter the text obtained by image recognition to obtain the keywords that need to be marked, and then use the operation guide text to mark the keywords in the page screenshot of each process node, so as to achieve the accurate marking of the content in the process node by using the operation guide text, and improve the accuracy of the operation guide. Therefore, the operation guide generation method based on image recognition proposed by the present invention can solve the problem of low accuracy when providing operation guidance to users.

[0127] like Figure 4 , which is a functional module diagram of an operation guide generating device based on image recognition provided by an embodiment of the present invention.

[0128] The image recognition-based operation guide generating device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the image recognition-based operation guide generating device 100 may include a text recognition module 101, a text segmentation module 102, a semantic extraction module 103, a text screening module 104 and a guide annotation module 105. The module of the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0129] In this embodiment, the functions of each module / unit are as follows:

[0130] The text recognition module 101 is used to obtain a page screenshot of each process node in the product purchase process and recognize the image text in the page screenshot;

[0131] The text segmentation module 102 is used to segment the image text to obtain text segmentation;

[0132] The semantic extraction module 103 is used to obtain the operation guide text of each process node in the product purchase process and extract the text semantics of the operation guide text;

[0133] The text screening module 104 is used to construct a decision tree model using the text semantics, and screen the text segmentation using the decision tree model to obtain text keywords;

[0134] The guidance marking module 105 is used to mark the position of the text keywords in the page screenshot using the operation guidance text, and to collect all the marked page screenshots into the interface operation guidance of the product purchase process.

[0135] In detail, each module described in the operation guide generating device 100 based on image recognition in the embodiment of the present invention is used in the same manner as described above. Figures 1 to 3 The same technical means as the operation guide generation method based on image recognition described in the above can produce the same technical effects, which will not be repeated here.

[0136] like Figure 5 , is a schematic diagram of the structure of an electronic device for implementing a method for generating an operation guide based on image recognition provided by an embodiment of the present invention.

[0137] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as an operation guide generation program based on image recognition.

[0138] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing an operation guide generation program based on image recognition, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0139] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the operation guide generation program based on image recognition, but also can be used to temporarily store data that has been output or is to be output.

[0140] The communication bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0141] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0142] Figure 5 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 5The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0143] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0144] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0145] The image recognition-based operation guide generation program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0146] Obtaining a page screenshot of each process node in the product purchase process, and identifying image text in the page screenshot;

[0147] Performing word segmentation on the image text to obtain text word segmentation;

[0148] Obtaining the operation guide text of each process node in the product purchase process, and extracting the text semantics of the operation guide text;

[0149] Using the text semantics to construct a decision tree model, and using the decision tree model to screen the text segmentation to obtain text keywords;

[0150] The operation guide text is used to mark the position of the text keyword in the page screenshot, and all the marked page screenshots are collected as the interface operation guide of the product purchase process.

[0151] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0152] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0153] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0154] Obtaining a page screenshot of each process node in the product purchase process, and identifying image text in the page screenshot;

[0155] Performing word segmentation on the image text to obtain text word segmentation;

[0156] Obtaining the operation guide text of each process node in the product purchase process, and extracting the text semantics of the operation guide text;

[0157] Using the text semantics to construct a decision tree model, and using the decision tree model to screen the text segmentation to obtain text keywords;

[0158] The operation guide text is used to mark the position of the text keyword in the page screenshot, and all the marked page screenshots are collected as the interface operation guide of the product purchase process.

[0159] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0160] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0162] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0163] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0164] The blockchain referred to in the present invention is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0165] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0166] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for generating operation instructions based on image recognition, characterized in that: The method comprises: Obtaining a page screenshot of each process node in the product purchase process, and identifying image text in the page screenshot; Performing word segmentation on the image text to obtain text word segmentation; Obtaining the operation guide text of each process node in the product purchase process, and extracting the text semantics of the operation guide text; Selecting one of the process nodes in the product purchase process as the target node one by one, assigning a value to a preset decision function using the text semantics of the operation guidance text of the target node, and using the assigned decision function as a decision condition to generate a decision tree for the target node; A decision tree that aggregates all process nodes in the product purchase process is used as a decision tree model, and the text segmentation is screened using the decision tree model to obtain text keywords; The operation guide text is used to mark the position of the text keyword in the page screenshot, and all the marked page screenshots are collected as the interface operation guide of the product purchase process.

2. The method for generating operation instructions based on image recognition according to claim 1, characterized in that: The identifying the image text in the page screenshot includes: Dividing the page screenshot into image blocks to obtain a plurality of image blocks; Performing convolution and pooling processing on each of the multiple image blocks respectively to obtain image features; Calculating the distance value between the image feature and a preset text feature, and collecting image blocks whose distance values ​​are less than a preset distance threshold as text image blocks; Binarizing the text image block, and dividing the binarized text image block into text blocks to obtain a single-word image; The single-word image is subjected to text recognition using a preset text recognition model, and the text recognition results are collected as graphic text.

3. The method for generating operation instructions based on image recognition according to claim 2, characterized in that: The binarization process of the text image block comprises: Selecting one pixel point from the text image block one by one as a target pixel point; Determine whether the pixel value of the target pixel is greater than a preset pixel threshold; If the pixel value of the target pixel point is greater than the pixel threshold, the pixel value of the target pixel point is set to a first parameter value; If the pixel value of the target pixel point is less than or equal to the pixel threshold, the pixel value of the target pixel point is set to a second parameter value.

4. The method for generating operation instructions based on image recognition according to claim 1, characterized in that: The step of segmenting the image text to obtain text segmentation includes: Segmenting the image text according to different preset lengths to obtain segmented texts; The segmented text is searched in a preset dictionary, and the segmented text retrieved in the preset dictionary is used as the text segmentation of the image text.

5. The method for generating operation instructions based on image recognition according to claim 1, characterized in that: The extracting the text semantics of the operation guide text includes: Segmenting the operation guide text to obtain operation guide segmentation words; Convert each word in the operation guide word segmentation into a word segmentation vector, and perform length unification processing on the word segmentation vector to obtain a uniform length vector; Each of the uniform-length vectors is used as a row vector to splice into a vector matrix, and the vector matrix is ​​used as the text semantics of the operation guidance text.

6. The method for generating operation instructions based on image recognition according to claim 5, characterized in that: The step of performing length unification processing on the word segmentation vector to obtain a uniform length vector includes: Counting the vector length of each of the word segmentation vectors, and determining the word segmentation vector with the largest vector length as the target vector; The vector length of each vector in the word segmentation vector is extended to the vector length of the target vector using preset parameters to obtain a uniform length vector.

7. An operation guide generation device based on image recognition, used to implement the operation guide generation method based on image recognition as claimed in any one of claims 1 to 6, characterized in that: The device comprises: A text recognition module is used to obtain a page screenshot of each process node in the product purchase process and recognize the image text in the page screenshot; A text segmentation module, used for segmenting the image text to obtain text segmentation; A semantic extraction module, used to obtain the operation guide text of each process node in the product purchase process, and extract the text semantics of the operation guide text; A text screening module, used to construct a decision tree model using the text semantics, and use the decision tree model to screen the text segmentation to obtain text keywords; The guidance marking module is used to mark the position of the text keywords in the page screenshot using the operation guidance text, and to collect all the marked page screenshots into the interface operation guidance of the product purchase process.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the operation guidance generation method based on image recognition as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating operation instructions based on image recognition as described in any one of claims 1 to 6 is implemented.

Images