Picture-based intent detection method and apparatus

Abstract

Embodiments of the present specification provide a picture-based intent detection method and device. For the technical scenario that the user question contains a page screenshot in the intelligent customer service scenario, on the one hand, the text semantic information and the text layout information are subjected to text coding to obtain text coding features related to the position of the text, and on the other hand, the region information in the page layout is coded to obtain image coding features and layout structure features. Then, the position information in the page layout structure is fused in combination with the page screenshot provided by the user, and the fused information is fused and coded with the text semantic information to obtain a page representation. Through the page representation, a candidate page matched with the page screenshot provided by the user can be identified. In this way, more rich page information can be used, and the accuracy of identifying the user intent can be improved.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a picture-based intent detection method and device. BACKGROUND

[0002] Image processing has a wide range of applications in daily production or life. For example: panorama segmentation, target recognition, text recognition, etc. In the field of artificial customer service, there may be a situation where a user uses a screenshot to assist in describing the problem encountered. In this case, the intelligent customer service needs to determine what business scenario the screenshot is generated in, what problem the screenshot may encounter, etc., so as to better serve the user and improve the user experience. SUMMARY

[0003] One or more embodiments of the present specification describe a picture-based intent detection method and device to solve one or more problems mentioned in the background.

[0004] According to a first aspect, a picture-based intent detection method is provided for detecting a business in a user problem based on a picture sent by the user, the method comprising: obtaining a first picture in a current user problem and first text information recognized for the first picture, the first text information comprising first text semantic information and first text position information; obtaining first text encoding features based on encoding of the first text information by a text encoding module; obtaining first image encoding features and first layout structure features by encoding a page layout of the first picture by an image encoding module; obtaining first position fusion features by mapping position information in the first layout structure features to a predetermined size via a position fusion module in combination with the first picture; obtaining a first page representation based on position information for the first picture by fusion encoding the first position fusion features, the first text encoding features, the first image encoding features, and the first layout structure features via a fusion encoding module; and determining a target page corresponding to the first picture based on the first page representation, and further determining a user intent.

[0005] In one embodiment, the first text semantic information comprises at least one text semantic unit, and the first text position information comprises coordinate information of each text box surrounding each text semantic unit.

[0006] In an embodiment, the first text semantic information comprises a first text corresponding to a first text box, the encoding of the first text information by the text encoding module comprises: representing each vertex of the first text box by a predetermined dimension of each position vector, wherein the characters in the first text share the position vectors of the first text box; embedding each position vector and a text semantic vector of the first text into a text feature vector of the first text as input data of the text encoding module; obtaining the first text encoding feature according to the processing of the text encoding module on the corresponding input data.

[0007] In an embodiment, the image encoding module is implemented by a convolutional neural network, and the output of the image encoding module is a plurality of feature maps of a predetermined size on a plurality of channels, a single feature point of the feature map corresponds to a single rectangular display area composed of a plurality of pixels on the first picture, and the first layout structure feature comprises vertex coordinates of the single rectangular display area on the first picture.

[0008] In an embodiment, the mapping of the position information in the first layout structure feature to a predetermined size via the position fusion module in combination with the first picture to obtain the first position fusion feature comprises: mapping the position information contained in the first layout structure feature and the first text feature to a predetermined size via the position fusion module based on the first picture, wherein the mapping result is related to the ratio between the first picture and the predetermined size; and determining the first position fusion feature according to the mapping result.

[0009] In an embodiment, the first position fusion feature, the first text encoding feature, the first image encoding feature and the first layout structure feature are fused and encoded via the fusion encoding module, so as to obtain the first page representation based on the position information for the first picture, which comprises: taking the first position fusion feature, the first image encoding feature and the first text encoding feature as input data of the fusion encoding module; and determining the first page representation according to the output of the fusion encoding module.

[0010] In an embodiment, the target page is selected from a plurality of candidate pages as page templates.

[0011] In an embodiment, each candidate page corresponds to a respective candidate representation obtained by previously processing the text recognition module, the text encoding module, the image encoding module, the position fusion module and the fusion encoding module; and the determination of the target page corresponding to the first picture based on the first page representation comprises: detecting the similarity between the first page representation and each candidate representation respectively; and selecting the target page from the candidate pages in the order from high to low similarity.

[0012] In an embodiment, the plurality of candidate pages comprises a first page, the first page corresponds to a first candidate representation, a similarity between the first page representation and the first candidate representation is described by a first similarity, the first similarity is determined based on one of cosine similarity, Jaccard coefficient, Euclidean distance, KL divergence, variance between the first page representation and the first candidate representation.

[0013] In an embodiment, the determining the target page corresponding to the first picture based on the first page representation comprises: inputting the first page representation into a pre-trained classification model or prediction model; and selecting the target page from the candidate pages according to an output result of the classification model or prediction model.

[0014] In an embodiment, the text encoding module, the image encoding module, the position fusion module, and the fusion encoding module are trained for page matching by training samples in a training set for a question and answer task, wherein each training sample picture in the training set corresponds to text position information extracted by a layout analyzer, page layout information, and a pre-labeled matched candidate page, the training sample picture comprises a second picture, and the matched candidate page of the second picture is a second page; the page matching training comprises: processing the second picture by using the text encoding module, the image encoding module, the position fusion module, and the fusion encoding module to obtain a second page representation; determining a first loss based on a correspondence between the second page representation and the second page; decoding the fusion encoding feature of the second picture based on the position information to obtain the text position information and the page layout information, and comparing the text position information and the page layout information with text position information and page layout information extracted by the layout analyzer for the second picture to determine a second loss; determining a current training loss based on the first loss and the second loss, and adjusting undetermined parameters of each module to reduce the current training loss.

[0015] In an embodiment, the second page corresponds to a second candidate representation obtained by processing by using the text encoding module, the image encoding module, the position fusion module, and the fusion encoding module; and determining the first loss based on the correspondence between the second page representation and the second page comprises: determining the first loss based on a first similarity between the second page representation and the second candidate representation, wherein the first loss is negatively correlated with the first similarity.

[0016] In an embodiment, the determining the first loss based on the correspondence between the second page representation and the second page comprises: inputting the second page representation into a pre-determined classification model or prediction model; and determining the first loss based on a comparison between an output result of the classification model or prediction model and a sample label determined by the second page.

[0017] In an embodiment, the text encoding module, the image encoding module, the position fusion module, and the fusion encoding module are further pre-trained based on each pre-training sample picture in a pre-training set before being trained by the training set, where each pre-training sample picture corresponds to text semantic information, text position information, and page layout information extracted by the layout analyzer; the pre-training sample picture includes a third picture, and the pre-training includes: obtaining the third picture; randomly obtaining current text information from the text semantic information; detecting whether the current text information is text information in the third picture; and based on the detection result, pre-training using the third picture.

[0018] In an embodiment, in a case where the current text information is not the text information in the third picture, the pre-training using the third picture based on the detection result includes: processing the third picture using the text encoding module, the image encoding module, the position fusion module, and the fusion encoding module to obtain a third page representation; classifying the third page representation using an auxiliary classifier to obtain an auxiliary classification result of whether the current text information and the third picture match; and comparing the auxiliary classification result with a sample label determined based on the detection result, to determine a third loss.

[0019] In an embodiment, in a case where the current text information is the text information in the third picture, the pre-training using the third picture based on the detection result includes: processing the current text information by the text encoding module to obtain text encoding features; performing text reconstruction on the text encoding features by a first mask language model to restore text hidden in the current text information in an encoding process as a target, to determine a fourth loss corresponding to the text encoding module; processing picture encoding features of the third picture and the text encoding features by the position fusion module to obtain fusion features of the third picture, the picture encoding features of the third picture being obtained by processing the third picture by the image encoding module; processing the fusion features of the third picture, the picture encoding features of the third picture, and the text encoding features by the fusion encoding module to obtain fusion encoding features of the third picture; performing text reconstruction on the fusion encoding features of the third picture by a second mask language model to restore text hidden in the current text information in an encoding process as a target, to determine a fifth loss; and adjusting pending parameters of each module to reduce a model loss determined based on the fourth loss and the fifth loss.

[0020] In an embodiment, the pre-training further comprises: decoding, by the auxiliary decoding module, the fusion encoded features of the third picture based on the position information to obtain the predicted text position information and the page layout information for the third picture; and comparing the predicted text position information and the page layout information for the third picture with the text position information and the page layout information extracted by the layout analyzer respectively to determine a sixth loss.

[0021] Then, the model loss determined based on the fourth loss and the fifth loss is determined according to a sum of the fourth loss, the fifth loss and the sixth loss.

[0022] According to a second aspect, there is provided an image-based intent detection apparatus for detecting a service targeted by a user question based on an image sent by the user, the apparatus comprising:

[0023] an acquisition unit configured to acquire a first image in a current user question and first text information recognized for the first image, the first text information comprising first text semantic information and first text position information;

[0024] a text encoding unit configured to obtain first encoded features based on encoding of the first text information by a text encoding module;

[0025] an image encoding unit configured to encode a page layout of the first image by an image encoding module to obtain first image encoded features and first layout structure features;

[0026] a position fusion unit configured to obtain first position fusion features by mapping position information in the first layout structure features to a predetermined size via a position fusion module in combination with the first image;

[0027] a picture representation unit configured to obtain first page representation based on position information for the first image by fusion encoding of the first position fusion features, the first text encoded features, the first image encoded features and the first layout structure features via a fusion encoding module;

[0028] an intent determination unit configured to determine a target page corresponding to the first image based on the first page representation and further determine a user intent.

[0029] According to a third aspect, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0030] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, characterized in that the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0031] Through the device and method provided by the embodiments of the present specification, in the case of artificial customer service, when the user describes the problem situation through the page picture mode, after obtaining the picture of the current user problem: on the one hand, the text semantic information and the text layout information are subjected to text coding to obtain the text coding features related to the position of the text, and on the other hand, the region information in the page layout is coded to obtain the image coding features and the layout structure features. Then, the position information in the page layout structure is fused in combination with the page screenshot provided by the user, and the fused information is fused and coded with the text semantic information to obtain the page representation. Through the page representation, the candidate page matched by the page screenshot provided by the user can be identified. Since the text semantic information, the text position information, the page layout information of the picture and the feature information of the picture itself are fused in the picture-based intent recognition process, the user intent can be recognized through more comprehensive picture information, and the accuracy of the recognition result is improved. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0033] Figure 1 is a schematic diagram of an application scenario under the technical concept of the present specification;

[0034] Figure 2 shows a schematic diagram of information content contained in a page screenshot according to the technical concept of the present specification;

[0035] Figure 3a shows a schematic diagram of the model architecture of the picture-based intent recognition of an embodiment of the present specification;

[0036] Figure 3b shows a schematic diagram of the model architecture of the picture-based intent recognition of another embodiment of the present specification;

[0037] Figure 4 shows a schematic diagram of the correspondence between the encoding coordinates and the pixels of a specific example of image coding;

[0038] Figure 5 shows a schematic diagram of the correspondence between the encoding coordinates and the pixels of a specific example of image coding; Figure 3a , Figure 3bA training diagram of each module in a model architecture for image-based intent recognition;

[0039] Figure 6a A diagram for image-based intent recognition according to one embodiment of the present specification is shown in Figure 3a 、 Figure 3b A pre-training flow diagram of each module in a model architecture for image-based intent recognition is shown in

[0040] Figure 6b A diagram for image-based intent recognition according to another embodiment of the present specification is shown in Figure 3a 、 Figure 3b A pre-training flow diagram of each module in a model architecture for image-based intent recognition is shown in

[0041] Figure 7 A diagram for image-based intent recognition according to one embodiment of the present specification is shown in

[0042] Figure 8 A diagram for image-based intent recognition according to one embodiment of the present specification is shown in DETAILED DESCRIPTION

[0043] The scheme provided by the present specification is directed to a business target recognition scene of an image. The scheme provided by the present specification will be described below in combination with the drawings. It should be noted that the technical scheme of the present application involves image processing, and some images or computer screenshots may be involved in the drawings. In order to make the diagram more clear, the color blocks of these images are not eliminated, and the clarity after conversion into a gray image does not affect the expression of the scheme.

[0044] Figure 1 An application scene example of the technical architecture of the present specification is shown. The application scene is an intelligent customer service (Q-A) scene. In the customer service scene, a user can consult a problem encountered through a page screenshot of a client. For example, as shown in Figure 1 , a user “Zhang San” can raise a problem (Q) of “abnormal display of a bill” through a page screenshot of a bill detail page. Figure 1 The server of the intelligent customer service can determine a candidate page corresponding to the page screenshot Figure 1 , and analyze from which link the page is connected, such as a “transfer → transaction bill → bill detail page” link connected to a “bill detail” page. According to the link information, the intelligent customer service can determine a possible candidate (such as a candidate question or a candidate answer), thereby providing a corresponding intelligent customer service reply answer (A) for the user.

[0045] In an intelligent customer service business, a user can feed back a problem encountered through a terminal, and the terminal transmits the corresponding problem of the user to a server through a network. The user problem can include a text description or picture information. In the case where the user problem includes a picture, the picture is usually a page screenshot of a problem. At this time, analyzing the source of the page where the picture is located helps to quickly identify the problem and improve the user experience brought by the intelligent customer service. Thus, the server can pre-set multiple candidate pages. These candidate pages can be template pages (also referred to as original sound pages) of various possible pages in the current application or the current business. In a specific business, various pages can be generated based on template pages, for example, elements of a form in a template page are assigned to business values under a specific business, thereby generating a specific page, and the like. Various problems can occur in a page operation or display process, and these problems can be determined by matching the page where the problem occurs to the corresponding template page.

[0046] Therefore, the page screenshot in the user problem can be associated with the corresponding template page based on image similarity by image matching between the page screenshot picture in the user problem and the template page as a candidate page. In this way, the technical problem of user intent recognition lies in similarity mining between the page screenshot picture provided by the user and the page image of the candidate page, or classifying the page screenshot picture provided by the user into the corresponding target category or predicting the probability of each classification category by taking each candidate page as each classification category.

[0047] Figure 1 The illustrated implementation scenario is illustrated by taking the matching between the page screenshot and the candidate page as an example. Taking a page screenshot as an example, Figure 1 In the specific implementation process, the page screenshot Figure 1 The matching process with each candidate page is an important step for the accuracy of the intelligent customer service reply. The page matching in the picture format usually includes matching of at least one of the following information: image texture information Image, text semantic information text, page layout Region, text layout detection, and the like. Among them, Region can describe the key area on the page, text is, for example, the text on the document recognized by the optical character recognition (OCR) technology, the area where the text on the document is recognized by the optical character recognition can be referred to as text layout detection information, and Image can represent the original picture with various layouts, i.e., image texture information, such as an app page screenshot, a document image containing layout information, and the like.

[0048] In order to more clearly describe the related information in the page screenshot, Figure 2 A specific example is shown. As Figure 2As shown, it is a page screenshot 200, and the entire page screenshot 200 can be referred to as an Image. The bounding box 201 is a function or a visually consistent area of the entire page, which describes the division of different display content areas of the page, belongs to the region layout Region category, and can be recorded as a Region box, which is used to describe the region layout in the page. The bounding box 202 corresponds to a text area, which describes the text position information in the page and belongs to the text layout detection category, so it can be used as a bounding box in the text layout, such as a detection box, which can be recorded as a detection box. The text 203 "store products" in the bounding box 202 is a complete semantic, which can be recorded as text information. Figure 2 In addition, there are other Region boxes, detection boxes, etc., which are not identified one by one. Among them, each Region box, detection box, etc. is, for example, an OCR (Optical Character Recognition) bounding box.

[0049] In the conventional technology, the page matching of the picture format can be performed by the following scheme:

[0050] 1) Document understanding model such as LayoutLM v1 / v2

[0051] This scheme often analyzes and processes image files after scanning text materials by means such as OCR, obtains text and layout information, and aligns the recognized text and the region. It is mainly used for processing document images. In addition to document images, page screenshots also have Region information, and LayoutLM cannot use Region information. Specifically, the input of LayoutLMv1 includes image features such as OCR text and OCR box, and an additional OCR image feature extraction model needs to be deployed online; the input of LayoutLMv2 includes OCR text and the entire picture Image, and end-to-end learning can be achieved, but only OCR text input and image input are considered, and at least Region information reflecting page layout is not considered.

[0052] 2) Multi-modal model represented by VIlBERT / Uniter

[0053] In this technical scheme, there is no spatial alignment relationship between image text and image. The input data usually includes image Region and image description text text, and does not consider individual picture Image and OCR input information detection. In addition, an additional Region feature extraction model is usually needed.

[0054] Therefore, the embodiment of the present specification provides a page representation scheme combining an original picture Image, text semantic information text, region layout (Region), and text layout (detection). Various information such as Image, text, Region, and detection are fused together and encoded into a page representation in the form of a tensor (such as a vector, a matrix, etc.), so as to obtain more accurate page representation results by using richer page information, thereby facilitating page matching or classification, etc. The picture recognition architecture combines region layout correction in the training process, so that the model has region layout recognition effect without additional layout region feature extraction model.

[0055] Firstly, referring to Figure 3a , the architecture of the online model of the embodiment of the present specification is introduced. As shown in Figure 3a , under the corresponding model layout, the model for processing the picture in the user question or the picture of the candidate page is at least structured by the following modules: a text encoding module, an image encoding module, a position fusion module, and a (position-based) encoding module. The functions of each module are introduced one by one as follows.

[0056] The text encoding module can encode the text information such as text semantic information text and text layout detection recognized in the page screenshot. The text semantic information text may, for example, be the recognized text content (the characters themselves), such as the information similar to the text 203 in Figure 2 . The text layout detection represents the position information of the region (such as the annotation box 202) where the text is located. The text information may, for example, be realized by an optical recognition model such as OCR or a layout analyzer, etc. The corresponding text information can be represented by characters and positions. Taking the text information as an example, its recognition result can be the characters themselves, for example, the recognized result of the text 203 in Figure 2 is “deposit product”. The text layout detection information can extract the position information of the annotation box. In an embodiment, the detection can be extracted as coordinate information. For example, the annotation box 202 in FIG. 2 can be represented by the four vertex coordinates of the rectangle corresponding to the annotation box 202 in the picture 200. In an embodiment, for convenience, the coordinates can be coordinate values in a coordinate system with the corner (such as the lower left corner) of the picture as the origin and pixels as the unit. In another embodiment, the vertex coordinates can be represented according to the row and column of the pixels. Assuming that the four vertex coordinates corresponding to the annotation box 202 are (200, 30), (200, 80), (220, 30), and (220, 80), in order to save data space, they can also be represented by the four coordinate values (200, 30, 220, 80).

[0057] The text encoding module can be implemented by a text processing model such as BERT Encoder, for example. The output of the text encoding module can contain both the semantics and the layout of the text, that is, the semantic information and the layout information of the text are fused by the text encoding module. For example, the coordinates of the four vertices of an OCR text box (such as the detection box 202 in Figure 2 ) are in the form of [x1, y1, x2, y2], and each character of the text in the text box can share the coordinates [x1, y1, x2, y2] of the OCR text box. In this way, the text semantics in the text box are closely combined with the text box position information.

[0058] The image encoding module is used to encode the features of the image to obtain the region layout features of the page (such as the features corresponding to the annotation box 201 in Figure 2 ). The image encoding module can be implemented by a model such as a convolutional neural network, such as by a ResNet Grid Encoder. The module can extract image texture features (corresponding to Image) and region features (corresponding to Region) from the page screenshot. Among them, the image texture features are used to describe the texture information of the image, and the region features are used to describe the corresponding region. Specifically, after feature extraction, a plurality of pixels in the page screenshot can be mapped to a region extraction box. Since the feature extraction is performed, the image is represented by a plurality of channel feature maps, and a single feature point on the feature map corresponds to a region extraction box. The values of the feature point in each channel can constitute a vector representation of the texture information in the corresponding region extraction box. As shown in Figure 4 , 3 is the final feature map, and one cell (feature point) in 3 corresponds to 4 cells (feature points) on 2, and each cell on 2 corresponds to 4 pixels on the initial picture 1. Therefore, one feature point on the final feature map can correspond to 16 pixels on the original picture. The 16 pixels correspond to a region extraction box. A single region extraction box can also be described by the relative coordinates of the corresponding four vertices such as [x1', y1', x2', y2'] and the like. Among them, x1', y1', x2', y2' are, for example, two sets of longitudinal and transverse coordinates obtained by combining the coordinate values of the four pixels at the vertices, such as the pixel row and pixel column where the four vertices correspond to the pixels.

[0059] To keep the consistency of the encoding results of pictures of various sizes, the position fusion module may, for example, draw on the idea of LayoutLM to map the coordinates of the text layout and the page layout to a unified space, such as a normalized 1000x1000 two-dimensional coordinate space, and then perform unified encoding. The position fusion module is configured to map the region layout Region based on the original picture Image to a two-dimensional space of a predetermined size. The predetermined size may be understood as a region of a predetermined size or an image of a predetermined size, etc. Through this step, regardless of the size of the original picture, the region information will be normalized to a unified size, thereby being more conducive to capturing the region information therein. The predetermined size is, for example, a two-dimensional coordinate space of 1000x1000, etc. This position normalization operation may be understood as fusion of position information, which may be referred to as Global2D-position Encoding.

[0060] In a possible design, the position fusion module not only normalizes the position information of the region extraction frame, but also normalizes the position information of the text (indicated by a dashed arrow in Figure 3a the figure).

[0061] In an embodiment, after EmbeddingLookUp by Tensorflow, the relevant vectors are superimposed as the encoding result. Tensorflow is a computational graph model, that is, a model that represents the operation process in the form of a graph. The Tensorflow program is generally divided into two stages of graph construction and graph execution. The graph construction stage is also called the graph definition stage, which defines the required operations in the graph model, and the result of each operation and the original input data can be called a node (operation, abbreviated as op). EmbeddingLookUp in Tensorflow can define an Embedding tensor (such as a matrix), and in the case of receiving an input, find the corresponding element in the defined tensor. In this way, specific to the position fusion module based on position, for example, one of the text encoding features or the image encoding features can be defined as an Embedding tensor (such as 1000x1000 dimensions), and the other as an input, and the corresponding element in the Embedding tensor is found for embedding. In this way, the position features in the picture can be fused based on position to form a fusion tensor.

[0062] Furthermore, the fusion encoding module, denoted as the position-enhance transformer encoder, is a position-enhanced encoding network. This module can perceive the positional relationship between the page layout and the text layout, and fuses and encodes page layout information, text layout information, text information, and image texture information using tensors. The fusion encoding module can be implemented, for example, through a transformer attention mechanism. The Transformer abandons traditional CNNs and RNNs; the entire network structure is composed entirely of attention mechanisms. More precisely, the Transformer consists of only self-attention and a Feed Forward Neural Network. The transformer attention mechanism can determine the spatial positional relationship of the input, such as the relationship between text position and page layout position, based on the position encoding results, and then encode accordingly.

[0063] Specifically, if the position of a text layout annotation box, after being normalized to a 1000×1000 range, is [800, 600, 900, 700], and the position of a page layout annotation box, after being normalized to a 1000×1000 range, is [600, 500, 990, 990], then the position encoding module can encode the text in the text box into the text in the annotation box of the page layout. In an optional embodiment, the positions of the text layout and the page layout are not direct position coordinates, but rather representation vectors encoded from the position coordinates. For example, [800, 600, 900, 700] is encoded into a 10-dimensional representation vector (a1, a2, ..., a...). 10 The values ​​[600, 500, 990, 990] are encoded as 10-dimensional representation vectors (b1, b2, ..., bb2). 10 The fusion encoding module can describe the positional relationship between detection boxes through the encoding results.

[0064] Figure 3a In the example, the output of the fusion encoding module is the image corresponding to the page (such as a screenshot of the page). Figure 1 The encoded representation integrates semantic text information, text layout information, page layout information, and image information of the page itself (such as color and pixel information). The encoded representation can be in the form of a one-dimensional tensor (vector), a two-dimensional tensor (matrix), or a three-dimensional tensor (multi-channel feature map), etc., and this specification does not limit it to this form.

[0065] Each candidate page can be pre-processed through a text recognition module, a text encoding module, an image encoding module, a location fusion module, and a fusion encoding module to obtain its respective candidate representation.

[0066] Thus, in combination with the specific scenario of Figure 1 , by comparing the encoding representation of the page screenshot Figure 1 with the respective candidate representation 1 to s corresponding to each candidate page, such as similarity detection, etc., one or more candidate representations closest to the encoding representation of the page screenshot Figure 1 can be selected, thereby determining one or more candidate pages corresponding to the page screenshot Figure 1 .

[0067] It is worth noting that Figure 1 the diagram shown is a schematic diagram of a specific application scenario, and in optional implementation manners, the candidate page corresponding to the page screenshot can also be determined in other ways through the page representation. For example, as shown in Figure 3b , the page representation is taken as the input of a classification model or a prediction model, and the candidate page corresponding to the page screenshot is determined according to the output result of the classification model or the prediction model. Specifically, in the classification scenario, the page screenshot Figure 1 can be classified into a candidate page through the classification model, and in the prediction scenario, the probability of the page screenshot Figure 1 corresponding to multiple candidate pages can be predicted, and optionally, the multiple candidate pages predicted can also be sorted in order of probability size.

[0068] Under the technical concept of the present specification, in order to ensure Figure 3a , Figure 3b the respective information extracted by each module in the model architecture shown can be trained through the assistance of other modules. In a specific business scenario, in order to determine that the encoding representation output by the fusion encoding module can accurately match the candidate representation of the candidate page that it actually corresponds to, further training of each module based on the samples annotated by the business task is also needed. The training concept can be: on the one hand, the encoding representation of the page screenshot and the candidate representation of the page picture of the candidate page corresponding to the page screenshot are as close as possible, while the encoding representations of other candidate pages are as different as possible; on the other hand, the page representation describes the region information and the text position information in the page screenshot as accurately as possible.

[0069] As shown in Figure 5 , the sample pictures in the training sample set can be manually annotated with corresponding candidate pages. The page representation (such as the second page representation) obtained by processing a single sample picture (such as the second picture) through the text encoding module, the image encoding module, the position fusion module, and the fusion encoding module should be as consistent as possible with the candidate representation (such as the second candidate representation) obtained by processing the corresponding candidate page (such as the second page). Thus, the page representation obtained by processing a single sample page screenshot can determine the task loss of each module in completing the related task, which can be referred to as the first loss.

[0070] According to a possible design, a comparison loss can be determined based on a comparison between the encoding representation of the sample picture and the candidate representation, such as Figure 5 The loss 1' is shown as the first loss. In the determination of the loss 1', a picture pair can be taken as a single training sample, and the single picture pair includes a sample picture and a page picture corresponding to a candidate page. In the case where the candidate page is the candidate page corresponding to the sample picture, it is equivalent to a positive sample, otherwise, it is equivalent to a negative sample. The two page representations corresponding to the picture pair in the positive sample are as close as possible (e.g., the similarity is as close to 1 as possible), while the two page representations corresponding to the picture pair in the negative sample are as far apart as possible (e.g., the similarity is as close to 0 as possible).

[0071] In addition, since the purpose of identifying the picture in the question and answer scenario is to match to the corresponding candidate page, and the change frequency of the candidate page is extremely small, in an optional embodiment, as shown in Figure 3a each candidate page is pre-processed using each module to obtain each candidate representation. In the model training, the sample picture is processed via each module to obtain the page representation of the page corresponding to the sample picture (e.g., the second page representation corresponding to the second picture), and compared with each candidate representation. In the case where the current candidate representation is the page representation of the candidate page corresponding to the sample picture, the first loss is determined with the goal that the page representation of the sample picture and the current candidate representation are as close as possible; in the case where the current candidate representation is not the page representation of the candidate page corresponding to the sample picture, the first loss is determined with the goal that the page representation of the sample picture and the current candidate representation are as far apart as possible.

[0072] According to another possible design, for the page representation of the sample picture, a candidate page to which the sample picture corresponds (at this time, each candidate page is respectively taken as a classification category) can also be predicted via processing of a classification module related to the relevant task, or a probability of the sample picture corresponding to each candidate page can also be predicted via processing of a prediction module. The classification module or the prediction module can be implemented, for example, by a fully connected neural network, a decision tree, or the like. At this time, a label of the classification model or the prediction model is determined according to a target of the business and the candidate page to which the sample picture corresponds. For example, based on the candidate page to which the sample picture corresponds, a label of a classification category of the sample picture is annotated, or a vector label of a classification probability of the sample picture classified into each classification category is annotated, and the annotation result is a sample label. Taking the classification module as an example, each classification category can be each candidate page, and a value output by the classification module corresponds to a corresponding candidate page category, for example, a value 1 corresponds to a bill detail page, a value 2 corresponds to an account login page, and the like. In the prediction module scenario, the page representation can be mapped as a probability of being classified into a corresponding each classification category, and a sample label thereof can be represented by a vector, each dimension of the vector respectively represents each candidate page, and a probability value of the candidate page corresponding to the sample image on a corresponding dimension of the label vector is 1, and a probability value of other candidate pages on a corresponding dimension is 0. Specifically, a more accurate encoding representation obtained by processing each sample picture by the trained modules is taken as input data of the classification module or the prediction module, the output result obtained by the classification module or the prediction module on the encoding representation is compared with the sample label determined based on the candidate page corresponding to the sample picture, and thus a first loss is determined.

[0073] Since the purpose of the position-based page representation is to represent as much as possible the text information, the text layout, the page layout, and the picture itself in the picture, so as to utilize more page information and improve the accuracy of the page representation, in order to avoid the influence of changes made to cater to the reduction of the loss 1' on the representation of the text layout, the page layout, and the like, the position information of the text annotation box and the page layout annotation box can also be adjusted simultaneously in the training process of each module. In this way, the text position information and the region position information in the page layout can be obtained by decoding the page representation, and then the decoding result is compared with the text annotation box and the Region annotation box obtained by OCR layout analysis, so as to determine a loss 2' (hereinafter referred to as a second loss).

[0074] Further, according to the technical concept of the present specification, the comprehensive loss of each module can be measured by superimposing the loss 1' and the loss 2' (i.e., the first loss and the second loss). In the actual training process, one batch of training samples can correspond to multiple sample pictures. At this time, the model loss can be the sum of the comprehensive losses of the sample pictures in the current batch.

[0075] Afterwards, the pending parameters of the classification module or the prediction module are adjusted in the direction of the model loss reduction, so as to train the respective module. In the training process, the pending parameters can also include the pending parameters in the decoding module, and in the specific scenario of the optional classification or prediction form (referring to the scenario architecture shown in Figure 3b the scenario architecture shown in the figure), the pending parameters in the involved classification module or prediction module can also be included.

[0076] The training process of the above modules focuses on the accuracy of the business task and the accuracy of the location information. However, the data processing process of each module also includes the processing of text semantic information. Therefore, according to the possible design, before each module is trained, it can also be pre-trained in relation to semantics. The pre-training process can have the accuracy of the text semantic information and the accuracy of the location information as dual goals. Specifically, the pre-training process can use training samples without artificial annotation (or without considering the business task) to preliminarily train each module, so that each module has a certain extraction ability for the corresponding information in the picture. And the training process can use training samples annotated based on related tasks, and on the basis of the pre-trained modules, each module is adaptively trained for specific tasks. In the training process, the sample labels of the annotated training samples are annotated as numerical values (such as numerical value 5 corresponding to the bill details page) corresponding to a single candidate page, or as vectors (such as a one-hot encoding vector of each candidate page) corresponding to multiple pages according to the output requirements. The pre-training process uses training samples that do not need to be annotated according to the business task, which can greatly reduce the annotation workload.

[0077] Figure 6a A specific implementation architecture of a pre-training process is shown. It can be understood that since the page can usually extract relevant structure and text information through the layout analyzer, in the pre-training process, the information of the training sample is obtained based on the picture itself. For the training sample used in the pre-training process, the text information text, the region layout information Region, and the text layout information detection can be extracted from the page layout analyzer (such as the layout extraction service in OCR, etc.).

[0078] Referring to Figure 6a As shown, the above modules are pre-trained by the sample pictures in the pre-trained page set. In the pre-training process, in order to enable each part to extract the corresponding features, the corresponding model loss can be determined in multiple steps. In the pre-training process, the input data includes two aspects: sample pictures and OCR text recognition results. The text recognition results can include all text recognition results for a sample picture.

[0079] For a sample picture (e.g., the third picture) in the pre-training set, the text recognition result (e.g., the OCR text and the text position information) is encoded by the text encoding module. The text encoding result can include the text information text and the text position information. To ensure that the encoding result restores the text information text as much as possible, the text encoding module can be determined by the encoding loss of the text semantics through text reconstruction by the masked language model (e.g., Hierachical MLM). The Hierachical MLM corrects the context words in the text through text reconstruction.

[0080] In the pre-training process, to reduce the recognition result of individual words on the overall text and the correct recovery ability of the text encoding, the text encoding loss can be determined based on the recovery situation of the masked part by masking part of the words in the text. Specifically, when encoding by the text encoding module, part of the words can be masked for encoding. The masked language model is equivalent to including a decoder, which can restore the encoding result of the text encoding module, and recover the masked words through text reconstruction in the restoration process. In this way, the text encoding loss is determined by detecting whether the relevant text can be correctly recovered according to the context. For example, for the text "transaction record details", the text "record" is masked for encoding, and then the masked language model tries to recover the text "record" according to the recovered context "transaction" and "details". The higher the accuracy of the recovered text, the smaller the text encoding loss, and vice versa. In Figure 6a In the pre-training process, to reduce the recognition result of individual words on the overall text and the correct recovery ability of the text encoding, the text encoding loss can be determined based on the recovery situation of the masked part by masking part of the words in the text. Specifically, when encoding by the text encoding module, part of the words can be masked for encoding. The masked language model is equivalent to including a decoder, which can restore the encoding result of the text encoding module, and recover the masked words through text reconstruction in the restoration process. In this way, the text encoding loss is determined by detecting whether the relevant text can be correctly recovered according to the context. For example, for the text "transaction record details", the text "record" is masked for encoding, and then the masked language model tries to recover the text "record" according to the recovered context "transaction" and "details". The higher the accuracy of the recovered text, the smaller the text encoding loss, and vice versa. In

[0081] On the other hand, the image encoding module can process the picture to obtain the page layout information, and the position fusion module can fuse and normalize the positions of the page layout based on the sample picture (e.g., the normalization method described above), so as to perceive the positional relationship between the page layout and the text layout. In optional embodiments, the text layout information and the page layout information can also be fused and normalized together (as shown by the dashed arrow in Figure 6a In the pre-training process, to reduce the recognition result of individual words on the overall text and the correct recovery ability of the text encoding, the text encoding loss can be determined based on the recovery situation of the masked part by masking part of the words in the text. Specifically, when encoding by the text encoding module, part of the words can be masked for encoding. The masked language model is equivalent to including a decoder, which can restore the encoding result of the text encoding module, and recover the masked words through text reconstruction in the restoration process. In this way, the text encoding loss is determined by detecting whether the relevant text can be correctly recovered according to the context. For example, for the text "transaction record details", the text "record" is masked for encoding, and then the masked language model tries to recover the text "record" according to the recovered context "transaction" and "details". The higher the accuracy of the recovered text, the smaller the text encoding loss, and vice versa. In

[0082] After the normalization of the position via the position fusion module, the fusion encoding module can also be used to combine the page layout information and the text encoding information for fusion encoding. Since the encoding here fuses the position information, in order to verify whether the text after fusing the position information can be recovered, a second mask model can be used for verification. Since the text encoding module encodes the text and masks part of the vocabulary, here the fusion encoding result is used to recover the text via the second mask model, and compared with the OCR text recognition result to verify the accuracy of the recovery.

[0083] The more accurate the prediction result is, the smaller the encoding loss based on the position is, and vice versa, the less accurate the prediction result is, the larger the encoding loss based on the position is. Figure 6a In this embodiment, the encoding loss based on the position can be denoted as loss 3 (hereinafter also referred to as the fifth loss). It can be understood that the loss 3 can measure the loss brought by the image encoding module, the text encoding module and the position fusion module at the same time.

[0084] For the fusion encoding result (such as the third page representation of the third picture), in order to verify the position accuracy of the position-based encoding result, a position-based decoding module (such as denoted as position-enhance transformer decoder) can also be set in the pre-training process, which can be realized through the attention mechanism of the transformer. The position-based decoding module can recover the position relationship encoded by the encoding module, so as to obtain the various annotation boxes of the text layout and the page layout, such as the annotation boxes 201, 203 in FIG. 2. On the other hand, the region annotation box (Region box) of the page layout and the text annotation box (OCR box) in the text layout can be pre-identified via an OCR or the like. Therefore, the page layout and the text layout annotation boxes pre-identified by the layout analyzer can be used to verify the position-based encoding and decoding process. Specifically, the various annotation boxes decoded by the decoding module can be compared with the annotation boxes identified by the OCR, and the more consistent the positions of the annotation boxes decoded by the decoding module are with the positions of the annotation boxes identified by the OCR, the smaller the encoding and decoding loss of the modules is, and vice versa. The loss here can be denoted as loss 4 (also referred to as the sixth loss). In fact, the loss 4 can reflect the overall loss of the image encoding module, the text encoding module, the position fusion module, the position-based overall encoding module and the decoding module.

[0085] In one embodiment, for one picture, or a batch of pictures, the loss 1, the loss 2, the loss 3 and the loss 4 can be calculated for each picture, and then the average of the loss 1, the loss 2, the loss 3 and the loss 4 of each picture can be calculated as the final loss of the picture or the batch of pictures. Figure 6aThe loss 2, loss 3, and loss 4 shown are superimposed as the model loss for the corresponding update round, and the undetermined parameters of the corresponding modules are adjusted in the direction of reducing the model loss, such as by adjusting the corresponding parameters through gradient descent, so as to pre-train each module in the model.

[0086] In some optional implementations, the encoding results of each module can be reverse-engineered for recognition. That is, if the sample image and the OCR text recognition result are inconsistent, it is desirable for each module to be able to distinguish the inconsistency. This ensures that the page representation obtained from encoding a user's page screenshot is as consistent as possible with the candidate representations of its corresponding candidate pages, and as inconsistent as possible with other candidate representations.

[0087] Therefore, after obtaining a sample image, an OCR text copy is obtained from the OCR recognition results of texts other than that sample image. For example... Figure 6b As shown, when the sample image and OCR text do not correspond, image-text matching (ITM) training can be performed on each module. In this case, the sample image and OCR text can be processed according to their respective modules, and the resulting page representation can be passed through an ITM classifier and mapped to a matching or non-matching category. This mapping result can be compared with the sample labels of the non-matching category to determine the loss 1 (also known as the third loss).

[0088] Optionally, for each image in the pre-training sample set, the OCR text and image recognized by the layout analyzer can be randomly selected. Here, since OCR can provide services such as text recognition and text annotation, the OCR recognition result of the layout analyzer is considered reliable. Specifically, if the selected OCR recognition result for any image (such as the third image) is the OCR text corresponding to that image, it is equivalent to constructing a positive sample. In this case, it can be processed according to... Figure 6a The execution architecture determines the loss as 1. Conversely, if the selected OCR text is not the text recognition result for the image, it is equivalent to constructing a negative sample. In this case, it can be done according to... Figure 6b The method shown determines the corresponding loss 1. Thus, in one training cycle, for this text recognition module, multiple positive and negative samples can be randomly selected and paired from the pre-training page set and the corresponding OCR text to construct a batch, thereby determining the model loss for the current training cycle. Further, optionally, in the sample images of the current batch, some sample images can obtain loss 1, while others can obtain loss 2, loss 3, and loss 4. Then, the comprehensive loss corresponding to this batch is the sum of the obtained loss 1, loss 2, loss 3, and loss 4.

[0089] In summary, in the model pre-training process, the text characters, text bounding boxes, and region bounding boxes of the page layout can be identified from the original picture in advance by the layout analyzer (such as the OCR layout analysis module), and these identification results are used as labels of the output results of the related modules in the above analysis process, so as to determine the model loss by supervising the output results of each module. In an optional embodiment, since the large OCR bounding box can reflect the text layout information of the graphic-text structure, the OCR bounding boxes can also be sorted according to the area size, and the regions ranked in the front (such as the top 15) are used as the labels of the annotation regions.

[0090] After the pre-training of the model, the text semantics, text layout, page layout, and other information in the picture can be preliminarily mined.

[0091] In this way, after the pre-training of the model, Figure 6a 、 Figure 6b The pre-training architecture shown in Figure 5 The text encoding module, the image encoding module, the position fusion module, and the fusion encoding module trained by the training architecture shown in can fuse the text semantic information, the text layout (such as the OCR detection box) information, the page layout (Region) information, and the original picture information in the picture. Since more comprehensive picture information is used, the page matching is more accurate.

[0092] The specific process of providing services by each module in the design concept of the specification will be described below in combination with a specific embodiment in the question and answer business scenario.

[0093] Please refer to Figure 7 , Figure 7 The intent detection flowchart based on the picture according to an embodiment is shown. The execution subject of the flowchart can be any computer, device, or server with certain computing power, such as a question and answer server. The flowchart is mainly used in the question and answer scenario of intelligent customer service, and is used to detect the business in the user question based on the picture sent by the user. Among them, at least the text encoding module, the image encoding module, the position fusion module, and the fusion encoding module shown in Figure 3a 、 Figure 3b are deployed on the question and answer server.

[0094] As shown in Figure 7As shown, an embodiment of this specification provides an image-based intent detection process including: Step 701, acquiring a first image in the current user question and first text information identified for the first image, the first text information including first text semantic information and first text location information; Step 702, obtaining a first text encoding feature based on the encoding of the first text information by a text encoding module; Step 703, encoding the page layout of the first image using an image encoding module to obtain a first image encoding feature and a first layout structure feature; Step 704, combining the first image, and mapping the location information in the first layout structure feature to a predetermined size via a location fusion module to obtain a first location fusion feature; Step 705, fusing and encoding the first location fusion feature, the first text encoding feature, the first image encoding feature, and the first layout structure feature via a fusion encoding module to obtain a first page representation based on location for the first image; Step 706, determining the target page corresponding to the first image based on the first page representation, and thus determining the user intent.

[0095] First, in step 701, the first image in the current user question and the first text information identified from the first image are obtained.

[0096] The current user question can be a question the user is inquiring about regarding a specific service, such as an inquiry about account switching or account statement details. During the user inquiry, to more clearly describe the problem arising under the corresponding service, images can be attached to the problem description. One or more images can be attached to an inquiry regarding a specific service. Any one of these images is referred to as the first image. The first image is usually a screenshot or snapshot of the page; in this manual, we collectively refer to it as a page screenshot.

[0097] For the first image, the server can match it to one or more candidate pages to associate it with related business functions, thereby inferring the user's intent or the problem the user is encountering. In an optional implementation, after identifying the business function associated with the first image, it can also combine it with text descriptions (such as...) Figure 1 The user's intent is determined by the phrase "My bill is not displaying correctly" in the context of the question. In step 701, after the user inputs the first image in the current question, the server can obtain the first image and perform text recognition on it to obtain the first text information. Text recognition can be performed using, for example, an OCR text recognizer or a layout analyzer. The first text information includes first text semantic information and first text position information. Here, "first" refers to the "first" in the "first image" and does not constitute a limitation on the substantive content.

[0098] The first text semantic information is the text information described above, for example Figure 2 The characters "deposit product", "confirm the amount", and the like in the illustrated page screenshot. The first text position information corresponds to the text layout detection described above, which can be the position where the text information is located.

[0099] Generally, the text can be divided into units according to the spacing between characters, arrangement, and the like, for example Figure 2 In the above, the characters in the text "deposit product" have small spacing, which can be a text semantic unit, while the text "payment method" and the text "deposit product" have large spacing, which can be different text semantic units. In the text information recognition process, the recognition of individual characters usually does not easily capture the semantic information therein, and therefore, in this specification, recognition can be performed according to text units. The text semantic information can be the recognized characters themselves, which can be divided into several character units according to the spacing. The text position information can be determined according to the bounding box position coordinates of the text semantic unit. For example, the text semantic unit is recognized in the page screenshot, and the start and end pixel rows and the start and end pixel columns are determined.

[0100] As an example, a single text semantic unit can usually be represented by four coordinates, which are the combined vertical and horizontal coordinates of the four vertex coordinates of the four vertices. The bounding box of a text semantic unit can be referred to as a text box or an OCR box. The four vertex coordinates of a bounding box can be, for example, [x1, y1], [x1, y2], [x2, y1], [x2, y2], which are combined as [x1, y1, x2, y2]. In some implementations, the encoding of a text box can also be described by the vertex coordinates of the bounding box and the size of the bounding box, such as [x1, y1, x2, y2, x2-x1, y2-y1]. At this time, the encoding result of a single text box can be a vector representation of six vectors.

[0101] Then, in step 702, based on the encoding of the first text information by the text encoding module, first text encoding features are obtained.

[0102] The text encoding can express the text information by numerical values on one hand, and can assign the position characteristics to the text semantic information on the other hand. The text encoding module is a semantic encoding network such as a Bert Encoder. Specifically, the text encoding module can encode the position information of the text in a single text semantic unit as the position information of the corresponding detection box (such as an OCR box). Taking the text encoding module as the Bert Encoder, the position information of the detection box can be embedded into the text feature vector as the input of the text encoding network. The Bert Encoder encodes the text by taking the word vector and the position vector as the embedding vector, and the position vector can be the embedding vector corresponding to the position coordinates of the text.

[0103] Further, each vertex of a bounding box can be represented by a position vector of a predetermined dimension (such as 10 dimensions). When embedding the position information into the text feature vector, the position vectors of the four vertices can be embedded into the text feature vector. It is worth noting that each character in a text semantic unit can use the position information corresponding to the text unit. Specifically, assuming that the first text semantic information includes a first text, the first text corresponds to a first text box, and before the first text is input into the text encoding information, the text semantic vector (such as the word vector) can be determined by tokenization, obtaining the word vector, etc. The text feature vector input into the text encoding module can be an embedding vector of the text semantic vector, the word position vector, the sentence vector, etc. In order to encode the position characteristics of the text, in this step 702, the position vector of the embedded text in the first picture can be used to replace the word position vector. For the first text, the position vectors of the first text box can be embedded. The position vectors of the first text box are the position vectors of a predetermined dimension corresponding to each vertex.

[0104] On the other hand, in step 703, the image encoding module is used to encode the page layout of the first picture to obtain the first image encoding feature and the first layout structure feature.

[0105] The image encoding module is used to extract the page layout feature. Specifically, each display area in the first picture can be described by position information. One display area can be a display area divided according to the similarity of functions or information in the page layout, such as Figure 2The bounding box 201 in the image is taken as a display region of a bill item. The information in one display region usually has relevance. The image encoding module is implemented by a structure such as ResNet (Residual Network). It can be understood that, in the case where the image encoding module is implemented via a convolutional neural network, the output thereof can include a plurality of region grids. A single region grid corresponds to a rectangular region (corresponding to a plurality of pixels) on the first picture, and corresponds to feature values on a plurality of channels. The plurality of feature values can be used to describe the encoding features of the rectangular region on the first picture corresponding to the grid, and meanwhile, the vertex coordinates of the plurality of pixels on the first picture to which the grid is mapped describe the layout structure features of the display region extracted for the first picture.

[0106] As a specific example, a rectangular bounding box can be represented by 4 coordinates, which are the combined longitudinal and lateral coordinates of the four vertices. For example, the four vertex coordinates of a bounding box are [x1', y1'], [x1', y2'], [x2', y1'], and [x2', y2'], which are combined as [x1', y1', x2', y2']. Since the receptive field in the output of one layer of a convolutional neural network is usually uniform, the number of pixels corresponding to each feature point on the original picture (here, the first picture) is uniform, so that the pixel position on the original picture can be mapped according to the position of the feature point. That is, the position information of the feature point contains the position information of the extracted bounding box.

[0107] In particular, each pixel in a bounding box in the image encoding result corresponds to the coordinates of the bounding box, or in other words, the position encoding result of each pixel in a bounding box is the same.

[0108] Further, by step 704, in combination with the first picture, the mapping of the position information in the first layout structure feature to the predetermined size via the position fusion module obtains the first position fusion feature.

[0109] According to the concept of the present specification, the first text encoding feature contains the position information of the text layout, and the first image encoding feature contains the position information of the page layout. In order to fuse the position information of the text layout and the page layout together, the position relationship between the first text encoding feature and the first image encoding feature can be fused in overall consistency in combination with the first picture.

[0110] Specifically, the position information involved in the image encoding result (grid coordinates) and the text encoding result can be normalized to a predetermined size of a feature region, such as a 1000x1000 size feature region, so that the mapping result of each coordinate is within the range of [0, 1000] regardless of the actual number of pixels of the first picture. Since the coordinate values in the image encoding result and the text encoding result are represented by vectors, the vectors can be mapped to integer values within the range of [0, 1000], so as to be regarded as normalized coordinates.

[0111] It can be understood that in the normalization process, a calculation result of a decimal number can occur, and a unit in the feature region can be regarded as a pixel, so that an upward rounding method can be used for mapping. That is, when the mapping coordinate is a decimal number, the larger integer is taken as the mapping coordinate value.

[0112] As a specific example, each coordinate in [x1, y1, x2, y2], [x1', y1', x2', y2'] is normalized to an integer value within 0-1000. In the normalization process, the 1000x1000 grid in the two-dimensional coordinate space can be regarded as a grid, and since the picture size is different, the size of each grid mapped to the picture itself is also different, or the corresponding pixel tree is also different. After dividing the grid, according to the position encoding result of the image encoding module (such as ResNet Grid) and the position encoding result of the text encoding module (such as the OCR bounding box), the position coordinate value of each position coordinate within the range of [0, 1000] after normalization is recalculated. The format of a single bounding box is, for example, 6 elements [x1, y1, x2, y2, width(x2-x1), height(y2-y1)], after EmbeddingLookUp, the 6 elements correspond to 6 vectors respectively, and the 6 vectors are fused, such as addition, to obtain a vector as the fusion result.

[0113] Then, the text layout (such as the OCR box) and the page layout (such as the Region bounding box) can be encoded respectively according to the normalized coordinate values. The encoding is the process of representing the coordinates by vectors. Similarly to the text encoding module and the image encoding module described above, the four vertices of a bounding box (OCR box or Region bounding box) can be embedded as 4 vectors or 6 vectors. Then, the vectors of the same bounding box are embedded as the corresponding encoding results. The embedding method is, for example, averaging, weighted averaging, weighted summation, taking the median, etc.

[0114] The normalized position encoding corresponds to the OCR box corresponding to the text layout detection and the region annotation box corresponding to the page layout Region, and is fused based on the first image. The fusion result can be referred to as a first position fusion feature.

[0115] In particular, it is found through experiments that, in step 704, the mapping of the position information in the first layout structure feature to the predetermined size via the position fusion module obtains the first position fusion feature, and the architecture without fusing the first text encoding feature has similar accuracy to the architecture of simultaneously mapping the first layout feature and the first text encoding feature to the predetermined size for fusion. Therefore, in step 704, the first position fusion feature can be obtained based on the first image via the mapping of at least the position information in the first layout structure feature to the predetermined size via the position fusion module.

[0116] Then, in step 705, the first position fusion feature, the first text encoding feature, the first image encoding feature, and the first layout structure feature are fused by the fusion encoding module to obtain a first page representation based on position information for the first image.

[0117] The first page representation is used to represent the first page, which can be, for example, Figure 3a 、 Figure 3b 、 Figure 5 、 Figure 6a 、 Figure 6b The encoding representation shown in the foregoing. In order to make the encoding of the fusion encoding module reflect the image information Image, the text semantic information text, the text layout detection, and the information of the page layout Region, the position fusion result in step 704 and the text encoding result (the first text encoding result, containing the text semantic information) in step 702 and the image encoding result (the first image encoding result, containing the texture information) in step 703 can be used as input data of the fusion encoding module.

[0118] The fusion encoding module can be implemented by a fully connected neural network, a neural network with an attention mechanism, etc. For example, in order to better represent the first image, an attention network with a transformer mechanism can be used, which will not be described here. The first page representation can be in the form of a tensor, such as a one-dimensional tensor (vector), a two-dimensional tensor (representation matrix), etc., which will not be described here.

[0119] Then, in step 706, the target page corresponding to the first image is determined based on the first page representation, and the user intent is further determined.

[0120] According to the foregoing principle description, there are two implementation manners of this step.

[0121] In an optional implementation, the consistency between the first page representation and each candidate representation corresponding to each candidate page is detected. In a popular way, the first page representation is most consistent with which candidate representation, and the candidate page corresponding to the candidate representation is determined.

[0122] At this time, the essence of step 706 is to determine the similarity between the first page representation and each candidate representation. It can be determined by the number of matching values, vector similarity, etc. Among them, the similarity between two representations can be determined by similarity determination methods such as cosine similarity, Jaccard coefficient, Euclidean distance, KL divergence, variance, etc. Taking cosine similarity as an example, assuming that the first page representation is denoted as vector a and a certain candidate representation is denoted as vector b, the cosine similarity between them can be denoted as The greater the cosine similarity, the higher the similarity between the two. It is worth noting that in the case of page representation by higher-dimensional tensor representation, the corresponding tensor can be flattened (for example, spliced by row / column) into a vector form to determine the similarity. The number of matching values is usually a one-to-one comparison of the values between the two representations. The more the same values, the higher the similarity.

[0123] According to another optional implementation, the process of step 706 to determine the candidate page corresponding to the first picture can also be implemented by a classification module or a prediction module. At this time, the first page representation can be input into a pre-trained classification module or prediction module, and the candidate page corresponding to the first picture can be determined according to the output result of the classification module or the prediction module, for example, the candidate page corresponding to the output value or the candidate page arranged in the front of the probability value is determined as the candidate page corresponding to the first picture, which will not be described here.

[0124] According to the candidate page corresponding to the first picture, other pages linked to the candidate page or other information input by the user in the current question (such as Figure 1 "my bill display is not normal" in the above example) can be further combined to determine the target question of the user, so as to identify the user's intention.

[0125] In view of the above process, for the technical scenario that the user question contains a page screenshot in the intelligent customer service scenario, a technical solution for recognizing the page screenshot provided by the user is proposed. Specifically, on the one hand, the text semantic information text and the text layout information detection in the page screenshot are extracted through the text recognition module OCR, and the text coding feature related to the position of the text is obtained through text coding. On the other hand, the region information in the page layout Region is coded through the image coding module to obtain the image coding feature (containing the Image texture information) and the layout structure feature (containing the Region position information). Then, the position information in the fused page layout is mapped to a predetermined size based on the position fusion module in combination with the page screenshot Image provided by the user, and the fused position information is fused and coded with the text semantic information text and the image coding information through the fusion coding module to obtain the page representation combined with the page screenshot Image, the text semantic information text, the text layout information detection and the page layout Region. The candidate page matched with the page screenshot provided by the user can be recognized through the page representation. Since more rich page information is used, the obtained page representation is more accurate, thereby improving the accuracy of recognizing the user intent.

[0126] According to another aspect, the embodiments of the present specification also provide a picture-based intent detection apparatus. The apparatus can be arranged in an intelligent customer service server, and is configured to detect a service to which a user question is directed based on a picture sent by the user. Figure 8 A picture-based intent detection apparatus 800 of an embodiment is shown. As shown, the apparatus 800 includes: Figure 8

[0127] An acquisition unit 801 configured to acquire a first picture in a current user question and first text information recognized for the first picture, the first text information including first text semantic information and first text position information;

[0128] A text coding unit 802 configured to obtain first coding features based on coding of the text coding module on the first text information;

[0129] An image coding unit 803 configured to code a page layout of the first picture using the image coding module to obtain first image coding features and first layout structure features;

[0130] A position fusion unit 804 configured to obtain first position fusion features by mapping position information in the first layout structure features to a predetermined size via the position fusion module in combination with the first picture;

[0131] ​The picture representation unit 805 is configured to perform fusion coding on the first position fusion feature, the first text coding feature, the first image coding feature and the first layout structure feature via a fusion coding module, so as to obtain a first page representation based on position information for the first picture.

[0132] The comparison unit 806 is configured to determine the target page corresponding to the first picture based on the first page representation, and further determine the user intention.

[0133] It is worth noting that, Figure 8 The apparatus embodiment shown corresponds to the method embodiment shown, and thus, in the corresponding scenario, the corresponding description of Figure 7 the apparatus embodiment is applicable to Figure 7 the corresponding scenario, and the corresponding description of Figure 8 the method embodiment is applicable to the corresponding scenario. Here, no further elaboration is made.

[0134] According to an embodiment of still another aspect, there is also provided a computer readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method described above. Figure 7 According to an embodiment of still another aspect, there is also provided a computer readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method described above.

[0135] According to an embodiment of still another aspect, there is also provided a computer device comprising a memory and a processor, the memory having stored thereon executable code that, when executed by the processor, realizes the method described above. Figure 7 According to an embodiment of still another aspect, there is also provided a computer device comprising a memory and a processor, the memory having stored thereon executable code that, when executed by the processor, realizes the method described above.

[0136] Those skilled in the art should be aware that, in one or more examples described above, the functions described in the embodiments of the present specification can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, these functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0137] The above detailed description of the specific implementation of the present specification, the purpose, technical solutions and beneficial effects of the technical concept of the present specification are further described in detail. It should be understood that the above is only a specific implementation of the technical concept of the present specification, and does not limit the protection scope of the technical concept of the present specification. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the present specification shall be included in the protection scope of the technical concept of the present specification.

Claims

1. A picture-based intent detection method for detecting a service targeted in a user question based on a picture sent by the user, the method comprising: obtaining a first picture in a current user question and first text information recognized for the first picture, the first text information comprising first text semantic information and first text position information; encoding the first text information based on a text encoding module to obtain first text encoding features; encoding a page layout of the first picture based on an image encoding module to obtain first image encoding features and first layout structure features; combining the first picture, mapping position information in the first layout structure features to a predetermined size via a position fusion module to obtain first position fusion features; fusing the first position fusion features, the first text encoding features, the first image encoding features and the first layout structure features via a fusion encoding module to obtain first page representations based on position information for the first picture; and determining a target page corresponding to the first picture based on the first page representations, and further determining a user intent.

2. The method of claim 1, wherein, The first text semantic information comprises at least one text semantic unit, and the first text position information comprises coordinate information of each text box surrounding each text semantic unit.

3. The method of claim 2, wherein, The first text semantic information comprises a first text, and the first text corresponds to a first text box. Each vertex of the first text box is represented by a predetermined dimension position vector, and characters in the first text share the position vector of the first text box. Each position vector and a text semantic vector of the first text are embedded into a text feature vector of the first text as input data of the text encoding module. The first text encoding features are obtained according to processing of the corresponding input data by the text encoding module.

4. The method of claim 1, wherein, The image encoding module is implemented by a convolutional neural network, and its output is a plurality of feature maps of a predetermined size on a plurality of channels, a single feature point of the feature map corresponds to a single rectangular display area composed of a plurality of pixels on the first picture, and the first layout structure features comprise vertex coordinates of the single rectangular display area on the first picture.

5. The method of claim 1, wherein, The combining the first picture, mapping position information in the first layout structure features to a predetermined size via a position fusion module to obtain first position fusion features comprises: mapping position information contained in the first layout structure features and the first text features to a predetermined size via a position fusion module based on the first picture, wherein the mapping result is related to a ratio between the first picture and the predetermined size; and determining the first position fusion features according to the mapping result.

6. The method of claim 1, wherein, The fusing the first position fusion features, the first text encoding features, the first image encoding features and the first layout structure features via a fusion encoding module to obtain first page representations based on position information for the first picture comprises: The first position fusion feature, the first image coding feature, and the first text coding feature are taken as input data of the fusion coding module; The first page representation is determined according to an output of the fusion coding module.

7. The method of claim 1, wherein, The target page is selected from a plurality of candidate pages as a page template.

8. The method of claim 7, wherein, Each candidate page corresponds to a respective candidate representation obtained by processing via a text coding module, an image coding module, a position fusion module, and a fusion coding module; and the target page corresponding to the first picture is determined based on the first page representation, including: Respective similarities between the first page representation and each candidate representation are detected; The target page is selected from the plurality of candidate pages in order from high to low similarity.

9. The method of claim 8, wherein, The plurality of candidate pages includes a first page corresponding to a first candidate representation, and a similarity between the first page representation and the first candidate representation is described by a first similarity degree determined based on one of a cosine similarity, a Jaccard coefficient, an Euclidean distance, a KL divergence, and a variance between the first page representation and the first candidate representation.

10. The method of claim 7, wherein, The target page corresponding to the first picture is determined based on the first page representation, including: The first page representation is input into a pre-trained classification model or prediction model; The target page is selected from the plurality of candidate pages according to an output of the classification model or prediction model.

11. The method of claim 1, wherein, The text coding module, the image coding module, the position fusion module, and the fusion coding module are trained for page matching by training samples in a training set for a question and answer task, wherein each training sample picture in the training set corresponds to text position information and page layout information extracted by a layout analyzer, and a pre-labeled matched candidate page, the training sample picture includes a second picture, and the matched candidate page of the second picture is a second page; the page matching training includes: The second picture is processed by the text coding module, the image coding module, the position fusion module, and the fusion coding module to obtain a second page representation; A first loss is determined based on a correspondence between the second page representation and the second page; A second loss is determined by decoding a fusion coding feature of the second picture based on position information to obtain text position information and page layout information, and comparing the text position information and the page layout information with text position information and page layout information extracted by the layout analyzer for the second picture; A current training loss is determined based on the first loss and the second loss, and a to-be-determined parameter of each module is adjusted to reduce the current training loss.

12. The method of claim 11, wherein, The second page corresponds to a second candidate representation obtained by processing via the text coding module, the image coding module, the position fusion module, and the fusion coding module; The first loss is determined based on a correspondence between the second page representation and the second page, including: The first loss is determined based on a first similarity between the second page representation and the second candidate representation, wherein the first loss is negatively correlated with the first similarity.

13. The method of claim 11, wherein, The determining the first loss based on the correspondence between the second page representation and the second page comprises: inputting the second page representation into a predetermined classification model or prediction model; determining the first loss according to a comparison between an output result of the classification model or prediction model and a sample label determined by the second page.

14. The method of any one of claims 11-13, wherein, The text encoding module, the image encoding module, the position fusion module, and the fusion encoding module are further pre-trained based on each pre-training sample picture in a pre-training set before being trained by the training set, wherein each pre-training sample picture corresponds to text semantic information, text position information, and page layout information extracted by a layout analyzer; the pre-training sample picture includes a third picture, and the pre-training comprises: obtaining the third picture; randomly obtaining current text information from the text semantic information; detecting whether the current text information is text information in the third picture; based on the detection result, pre-training using the third picture.

15. The method of claim 14, wherein, In a case where the current text information is not the text information in the third picture, the pre-training using the current picture based on the detection result comprises: processing the third picture using the text encoding module, the image encoding module, the position fusion module, and the fusion encoding module to obtain a third page representation; classifying the third page representation using an auxiliary classifier to obtain an auxiliary classification result of whether the current text information and the third picture match; comparing the auxiliary classification result with a sample label determined based on the detection result to determine a third loss.

16. The method of claim 14, wherein, In a case where the current text information is the text information in the third picture, the pre-training using the current picture based on the detection result comprises: processing the current text information by the text encoding module to obtain text encoding features; text reconstruction by a first mask language model on the text encoding features to restore text hidden in the encoding process of the current text information as a target to determine a fourth loss corresponding to the text encoding module; processing picture encoding features of the third picture and the text encoding features by the position fusion module to obtain fusion features of the third picture, the picture encoding features of the third picture being obtained by processing the third picture by an image encoding module; processing the fusion features of the third picture, the picture encoding features of the third picture, and the text encoding features by a fusion encoding module to obtain fusion encoding features of the third picture; text reconstruction by a second mask language model on the fusion encoding features of the third picture as a target to restore text hidden in the encoding process of the current text information to determine a fifth loss; adjusting pending parameters of each module to reduce a model loss determined based on the fourth loss and the fifth loss.

17. The method of claim 16, wherein, The pre-training further comprises: decoding the fusion encoding features of the third picture based on position information by an auxiliary decoding module to obtain predicted text position information and page layout information for the third picture; comparing the text position information and the page layout information predicted for the third picture with the text position information and the page layout information extracted by the layout analyzer, to determine a sixth loss; wherein, the model loss determined based on the fourth loss and the fifth loss is determined according to a sum of the fourth loss, the fifth loss and the sixth loss. 18.A picture-based intent detection apparatus for detecting a service targeted in a user question based on a picture sent by the user, the apparatus comprising: an acquisition unit configured to acquire a first picture in a current user question and first text information recognized for the first picture, the first text information comprising first text semantic information and first text position information; a text encoding unit configured to obtain first encoding features based on encoding of the first text information by a text encoding module; an image encoding unit configured to encode a page layout of the first picture by an image encoding module to obtain first image encoding features and first layout structure features; a position fusion unit configured to obtain first position fusion features by mapping position information in the first layout structure features to a predetermined size via a position fusion module in combination with the first picture; a picture representation unit configured to obtain a first page representation based on position information for the first picture by fusion encoding of the first position fusion features, the first text encoding features, the first image encoding features and the first layout structure features via a fusion encoding module; an intent determination unit configured to determine a target page corresponding to the first picture based on the first page representation, and further determine a user intent. 19.A computer-readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-17.

20. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and the processor executes the executable codes to implement the method of any one of claims 1-17. The memory stores executable codes, and the processor executes the executable codes to implement the method of any one of claims 1-17.

Images