Image-text interaction method and device, electronic equipment and storage medium

Abstract

The invention provides an image-text interaction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target image; obtaining text information corresponding to the target image based on the label information of the target image; the label information is used for representing that the category of the target image is a text image or a non-text image; obtaining a first interaction instruction, wherein the first interaction instruction comprises at least one of a reading instruction, a translation instruction, a composition instruction or a text composition instruction; and in response to the first interaction instruction, obtaining an image-text interaction result of the target image based on the text information. Through the scheme provided by the invention, the text contained in the text image cannot be acquired, and the text information corresponding to the non-text image can also be acquired; furthermore, through the first interaction instruction, not only can the text contained in the text image be read, translated or patterned, but also the text information corresponding to the non-text image can be patterned, so that the image-text interaction experience of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of machine vision, and in particular to a method, device, electronic device and storage medium for image-text interaction. Background Art

[0002] With the rapid development of machine vision, various text-and-image interactive devices or corresponding text-and-image interactive systems such as smart phones, smart tablets, and computers have been widely used in various fields such as education and teaching, corporate meetings, and commercial exhibitions, bringing great convenience to people's study, work, and life. However, the text-and-image interactive systems in related technologies can only recognize the text contained in the text image when performing text-and-image interaction, but cannot further process the recognized text. Summary of the invention

[0003] The present disclosure provides a graphic-text interaction method, device, electronic device and storage medium to solve the problems in the related art.

[0004] A first aspect of the present disclosure provides a method for text-image interaction, the method comprising:

[0005] Get the target image;

[0006] Based on the label information of the target image, obtaining text information corresponding to the target image; the label information is used to characterize whether the category of the target image is a text image or a non-text image;

[0007] Obtaining a first interactive instruction, wherein the first interactive instruction includes at least one of a reading instruction, a translation instruction, a composition instruction, or a text composition instruction;

[0008] In response to the first interaction instruction, a graphic-text interaction result of the target image is obtained based on the text information.

[0009] In some embodiments of the present disclosure, the acquiring text information corresponding to the target image based on the label information of the target image includes:

[0010] Based on the label information of the target image, obtaining the category of the target image;

[0011] If the target image is a text image, obtaining the text contained in the text image;

[0012] If the target image is a non-text image, the text information corresponding to the non-text image is obtained.

[0013] In some embodiments of the present disclosure, the obtaining of the text contained in the text image includes:

[0014] Acquire text position information in the text image;

[0015] Based on the text position information in the text image, the text in the text image is acquired.

[0016] In some embodiments of the present disclosure, the obtaining text position information in the text image includes:

[0017] Obtaining text paragraph position information in the text image;

[0018] Segmenting the text paragraph position information to obtain text line position information in the text image;

[0019] The text line position information is segmented to obtain the single-word text position information in the text image.

[0020] In some embodiments of the present disclosure, acquiring the text in the text image based on the text position information includes:

[0021] Based on the single-word text position information in the text image, identifying each single-word text in the text image;

[0022] Based on each single-word text in the text image and its corresponding single-word text position information, the text in the text image is acquired.

[0023] In some embodiments of the present disclosure, the first interaction instruction includes a reading instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes:

[0024] Based on the reading instruction, obtaining target text information corresponding to the reading instruction;

[0025] In response to the reading instruction, the target text information is converted into audio information.

[0026] In some embodiments of the present disclosure, the first interaction instruction includes a translation instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes:

[0027] Based on the translation instruction, obtaining target text information corresponding to the translation instruction;

[0028] In response to the translation instruction, the target text information is converted into first translated text information.

[0029] In some embodiments of the present disclosure, the first interaction instruction includes a composition instruction, and the step of obtaining a text-image interaction result of the target image based on the text information in response to the first interaction instruction includes:

[0030] Based on the composition instruction, obtaining target text information corresponding to the composition instruction;

[0031] In response to the composition instruction, the target text information is converted into image information.

[0032] In some embodiments of the present disclosure, the first interaction instruction includes a text composition instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes:

[0033] Based on the composition instruction, the text information is converted into composition text.

[0034] In some embodiments of the present disclosure, the image-text interaction method further includes:

[0035] Obtaining a second interaction instruction;

[0036] In response to the second interactive instruction, the composition text is converted into second translated text information.

[0037] A second aspect of the present disclosure provides a graphic-text interaction device, the device comprising:

[0038] A first acquisition unit, used for acquiring a target image;

[0039] A second acquisition unit is used to acquire text information corresponding to the target image based on the label information of the target image; the label information is used to characterize whether the category of the target image is a text image or a non-text image;

[0040] A third acquisition unit is used to acquire a first interaction instruction, where the first interaction instruction is used to indicate a user's demand for graphic-text interaction with the target image;

[0041] A fourth acquisition unit is used to acquire a text-image interaction result of the target image based on the text information in response to the first interaction instruction.

[0042] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.

[0043] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0044] The fifth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.

[0045] The present disclosure provides a method, device, electronic device and storage medium for text-image interaction. The method includes: obtaining a target image; obtaining text information corresponding to the target image based on the label information of the target image; the label information is used to characterize the category of the target image as a text image or a non-text image; obtaining a first interaction instruction, the first interaction instruction including at least one of a reading instruction, a translation instruction, a composition instruction or a text composition instruction; in response to the first interaction instruction, obtaining a text-image interaction result of the target image based on the text information. Through the scheme provided by the present disclosure, the text contained in the text image cannot be obtained, but the text information corresponding to the non-text image can also be obtained; further, through the first interaction instruction, not only the text contained in the text image can be read, translated or composed, but also the text information corresponding to the non-text image can be composed, which improves the user's text-image interaction experience.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0048] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0049] Figure 2 A schematic diagram of the flow of the graphic-text interaction method provided in the embodiment of the present disclosure;

[0050] Figure 3 A flowchart of a graphic-text interaction method provided for the first application example of the present disclosure;

[0051] Figure 4 A flowchart of a graphic-text interaction method provided for the second application example of the present disclosure;

[0052] Figure 5 A graphic-text interaction effect diagram provided for the second application example of the present disclosure;

[0053] Figure 6A schematic diagram of a flowchart of a graphic-text interaction method provided for the third application example of the present disclosure;

[0054] Figure 7 A schematic diagram of the structure of a graphic-text interaction device provided in an embodiment of the present disclosure;

[0055] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0056] Fig. 9 A schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0058] With the rapid development of machine vision, various text-and-image interactive devices or corresponding text-and-image interactive systems such as smart phones, smart tablets, and computers have been widely used in various fields such as education and teaching, corporate meetings, and commercial exhibitions, bringing great convenience to people's study, work, and life. However, the text-and-image interactive systems in related technologies can only recognize the text contained in the text image when performing text-and-image interaction, but cannot further process the recognized text.

[0059] In order to solve the problems existing in the related technology, the present disclosure proposes a text-image interaction method, which can not only obtain the text contained in the text image, but also obtain the text information corresponding to the non-text image; further, through the first interaction instruction, not only can the text contained in the text image be read, translated or composed, but the text information corresponding to the non-text image can also be composed, thereby improving the user's text-image interaction experience.

[0060] Before introducing the detailed solution of the present disclosure, the scenario to which the solution of the present disclosure is applied is described first. Figure 1 FIG. 1 is an application scenario diagram of an image interaction method in an embodiment. Figure 1 As shown, the application scenario includes an electronic device 101, in which a camera module and several applications can be installed. The application can initiate an image acquisition instruction to obtain a target image, and the camera module acquires a target image 102. Among them, the camera module can include a front camera module and / or a rear camera module. Finally, the target image 102 is sent to the target application. Among them, the electronic device 101 can be a smart phone, a tablet computer, a personal digital assistant, a wearable device, etc.

[0061] In some optional embodiments, the electronic device 101 may also be a vehicle-mounted device or an Internet of Vehicles device, such as a smart car, etc. In the present disclosure, only a smart phone is taken as an example, but it does not limit the scope of the present disclosure.

[0062] A camera may be installed on the electronic device 101, and the target image 102 may be acquired through the installed camera. Several cameras may be installed on the electronic device 101, and the installation positions are not limited. For example, a camera may be installed on the front panel of the electronic device, and two cameras may be installed on the back panel. The camera may also be installed inside the electronic device in an embedded manner, and then the camera may be opened by rotating or sliding. Specifically, a front camera and a rear camera may be installed on the electronic device 101, and the front camera and the rear camera may acquire the target image 102 from different viewing angles.

[0063] It should be understood that in the present disclosure, the front camera or the rear camera is only used as an example to distinguish the shooting angles of different cameras, rather than to limit the functions of multiple cameras. The multiple cameras in the present disclosure can be rear cameras or front cameras at the same time, which is not limited in the present disclosure.

[0064] Several applications can be installed in the electronic device 101. An application refers to software written for a certain application purpose in the electronic device 101. The electronic device 101 can provide services for the user's needs through the application. When the application needs to obtain the target image 102, an image acquisition instruction will be initiated, and the electronic device will call the camera module to capture the target image 102 according to the image acquisition instruction. The image acquisition instruction refers to an instruction used to trigger the image acquisition operation.

[0065] The electronic device is also provided with a processor, and the image processing module in the processor can process the target image 102 captured by the camera module, for example, execute the image-text interaction method provided in the present disclosure.

[0066] The embodiments of the present disclosure can be applied to scenarios where image-text interaction is required, for example, image-text interaction in a camera-based literacy mode, image-text interaction in a camera-based translation mode, image-text interaction in a camera-based text composition mode, or image-text interaction in a camera-based composition mode.

[0067] Figure 2 A flowchart of a graphic-text interaction method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the image-text interaction method comprises the following steps:

[0068] Step 201, acquiring a target image;

[0069] In one embodiment, the method of acquiring the target image includes but is not limited to taking a photo, taking a screenshot, acquiring from a local gallery, or crawling from the network.

[0070] In one embodiment, while acquiring the target image, label information corresponding to the target image is also acquired, and the label information is used to characterize whether the category of the target image is a text image or a non-text image.

[0071] Step 202: based on the label information of the target image, obtain text information corresponding to the target image; the label information is used to characterize whether the category of the target image is a text image or a non-text image;

[0072] In one embodiment, the user's demand for text-image interaction generally includes converting text into images, or converting images into text. Whether converting text into images or converting images into text, it is necessary to obtain text information corresponding to the target image. For a text image, the corresponding text information is the text contained in the text image; for a non-text image, the corresponding text information is the image features of the non-text image, such as scenes and objects in the image.

[0073] Based on this, the step of obtaining text information corresponding to the target image based on the label information of the target image includes:

[0074] Based on the label information of the target image, obtaining the category of the target image;

[0075] If the target image is a text image, obtaining the text contained in the text image;

[0076] If the target image is a non-text image, the text information corresponding to the non-text image is obtained.

[0077] In one embodiment, the label information of the target image may be obtained through an input operation, and the input operation includes but is not limited to voice input, text input, etc.

[0078] In one embodiment, the method for obtaining the text contained in the text image may adopt a text recognition algorithm based on an optical character recognition (OCR) algorithm. The method for obtaining the text information corresponding to the non-text image may use a convolutional neural network (CNN) to perform target detection on the non-text image to obtain a feature vector of the non-text image; then the feature vector is input into a neural language model (NLP) to obtain the text information corresponding to the non-text image.

[0079] Step 203, obtaining a first interactive instruction, wherein the first interactive instruction includes at least one of a reading instruction, a translation instruction, a composition instruction, or a text composition instruction;

[0080] In one embodiment, a method of obtaining the first interaction instruction includes but is not limited to obtaining the first interaction instruction through a voice input operation or a text input operation.

[0081] In one embodiment, the first interaction instruction is used to indicate a user's demand for graphic and text interaction with the target image.

[0082] In one embodiment, the reading instruction refers to converting the text contained in the text image into audio. The translation instruction refers to translating the text contained in the text image into text corresponding to a specified language. The composition instruction refers to converting the text contained in the text image into a corresponding composition image. The composition instruction refers to converting the text information corresponding to the non-text image into a composition text.

[0083] Step 204: In response to the first interaction instruction, obtain a text-image interaction result of the target image based on the text information.

[0084] The present disclosure provides a method, device, electronic device and storage medium for text-image interaction. The method includes: obtaining a target image; obtaining text information corresponding to the target image based on the label information of the target image; the label information is used to characterize the category of the target image as a text image or a non-text image; obtaining a first interaction instruction, the first interaction instruction including at least one of a reading instruction, a translation instruction, a composition instruction or a text composition instruction; in response to the first interaction instruction, obtaining a text-image interaction result of the target image based on the text information. Through the scheme provided by the present disclosure, the text contained in the text image cannot be obtained, but the text information corresponding to the non-text image can also be obtained; further, through the first interaction instruction, not only the text contained in the text image can be read, translated or composed, but also the text information corresponding to the non-text image can be composed, which improves the user's text-image interaction experience.

[0085] In one embodiment, the method for obtaining the text contained in the text image may adopt a text recognition algorithm based on an OCR algorithm. The text recognition algorithm based on an OCR algorithm is divided into two steps, the first step is text detection, and the second step is text recognition.

[0086] Based on this, in one embodiment, obtaining the text contained in the text image includes:

[0087] Acquire text position information in the text image;

[0088] Based on the text position information in the text image, the text in the text image is acquired.

[0089] In one embodiment, the method for obtaining the text position information in the text image may adopt any one of a morphological operation method, a maximally stable extremal region (MSRE) algorithm, and a method based on a connectionist text proposal network (CTPN).

[0090] Taking the use of morphological operations to obtain the text position information in the text image as an example, the text is first binarized, and then based on the principle that the pixel values ​​of the text in the text image are roughly the same, the text position information in the text image is obtained using dilation and erosion operations.

[0091] Taking the use of the MSRE algorithm to obtain the text position information in the text image as an example, the text image is binarized, and the threshold value increases from 0 to 255. At this time, some connected domains in the image change very little or do not change, and the connected domain is called the maximum stable extremum region. Since the pixel values ​​of the text area in the text image are consistent, the maximum stable extremum region can be used as the text position information in the text image.

[0092] In one embodiment, since the text in the text image is usually recognized word by word, in order to ensure that the recognized single-word text is at the correct position in the text, the position information of the single-word text needs to be detected and aligned.

[0093] Based on this, in one embodiment, obtaining the text position information in the text image includes:

[0094] Obtaining text paragraph position information in the text image;

[0095] Segmenting the text paragraph position information to obtain text line position information in the text image;

[0096] The text line position information is segmented to obtain the single-word text position information in the text image.

[0097] In one embodiment, the method for obtaining the text paragraph position information in the text image may adopt any one of a morphological operation method, an MSRE algorithm, and a CTPN method.

[0098] In one embodiment, the text paragraph position information in the text image may be one or more text paragraph position information, or there may be no text paragraph position information, for example, the text image only contains one or more scattered single-word texts. Similarly, the text line position information may be one or more text line position information, or there may be no text line position information.

[0099] In one embodiment, after the text position information in the text image is detected by a text detection method, the text in the text image can be acquired by a text recognition method.

[0100] Based on this, in one embodiment, obtaining the text in the text image based on the text position information includes:

[0101] Based on the single-word text position information in the text image, identifying each single-word text in the text image;

[0102] Based on each single-word text in the text image and its corresponding single-word text position information, the text in the text image is acquired.

[0103] In one embodiment, based on the single-word text position information in the text image, a method for identifying each single-word text in the text image may adopt a template matching algorithm, a support vector machine (SVM) and a neural network algorithm.

[0104] Taking the template matching algorithm as an example, by calculating the similarity between the image corresponding to each single-word text position information and all preset sub-images, the preset sub-image refers to the standard single-word text image, and the text corresponding to the preset sub-image with the highest similarity is used as the single-word text recognition result.

[0105] Taking the neural network algorithm as an example, the features of single-word images are extracted, and then the extracted character feature information is used as the input of the neural network to train the network, and finally the trained network is used to recognize single-word text.

[0106] In one embodiment, the first interaction instruction includes a reading instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes:

[0107] Based on the reading instruction, obtaining target text information corresponding to the reading instruction;

[0108] In response to the reading instruction, the target text information is converted into audio information.

[0109] In one embodiment, the target text information corresponding to the reading instruction refers to part or all of the text information, because the user may only need to read part of the text contained in the text image.

[0110] In one embodiment, the first interaction instruction includes a translation instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes:

[0111] Based on the translation instruction, obtaining target text information corresponding to the translation instruction;

[0112] In response to the translation instruction, the target text information is converted into first translated text information.

[0113] In one embodiment, the target text information corresponding to the translation instruction refers to part or all of the text information, because the user may only need to translate part of the text contained in the text image.

[0114] In one embodiment, the first interaction instruction includes a composition instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes:

[0115] Based on the composition instruction, obtaining target text information corresponding to the composition instruction;

[0116] In response to the composition instruction, the target text information is converted into image information.

[0117] In one embodiment, the method for converting the target text information into image information may be based on a generative adversarial network method. Specifically, firstly, the target text information is converted into a word embedding using a word embedding technique; and then the word embedding is converted into image information using a generative adversarial network.

[0118] In one embodiment, the first interaction instruction includes a text composition instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes:

[0119] Based on the composition instruction, the text information is converted into composition text.

[0120] In one embodiment, the method of converting the text information into composed text may use CNN to perform target detection on the non-text image to obtain a feature vector of the non-text image; then input the feature vector into the NLP network to obtain the text information corresponding to the non-text image.

[0121] In one embodiment, the above-mentioned text may be further processed, such as translated.

[0122] Based on this, in one embodiment, the image-text interaction method further includes:

[0123] Obtaining a second interaction instruction;

[0124] In response to the second interactive instruction, the composition text is converted into second translation text information.

[0125] The interactive method provided by the embodiment of the present disclosure can not only obtain the text contained in the text image, but also obtain the text information corresponding to the non-text image; further, through the first interactive instruction, not only can the text contained in the text image be read, translated or composed, but the text information corresponding to the non-text image can also be composed, thereby improving the user's image-text interactive experience.

[0126] The graphic-text interaction method provided by the present disclosure is further illustrated below using the first application example.

[0127] like Figure 3 As shown, the graphic-text interaction method provided by the first application example of the present disclosure includes the following steps:

[0128] Step 301, acquiring a target image by taking a photo, wherein the target image is a text image;

[0129] Step 302, using OCR characters to recognize text information in the target image;

[0130] Step 303, obtaining a first interactive instruction through a trigger operation, wherein the first interactive instruction includes at least one of a reading instruction, a translation instruction, a composition instruction, and a copy instruction;

[0131] Step 304: In response to the first interaction instruction, obtain a text-image interaction result of the target image based on the text information.

[0132] The graphic-text interaction method provided by the present disclosure is further illustrated below with a second application example.

[0133] like Figure 4 As shown, the graphic-text interaction method provided by the second application example of the present disclosure includes the following steps:

[0134] Step 401, obtaining a target image, wherein the target image is a text image;

[0135] Step 402, using OCR characters to recognize text information in the target image;

[0136] Step 403, obtaining a first interaction instruction, where the first interaction instruction is a composition instruction;

[0137] Step 404: In response to the composition instruction, convert the target text information into image information.

[0138] like Figure 5 As shown, Figure 5 This is a picture of the text-graphic interaction effect provided by the second application example of the present disclosure. It can be seen that the text-graphic interaction method provided by the second application example of the present disclosure can be used to Figure 5 The text image on the middle left, which mainly contains the text of "Viewing the Waterfall at Mount Lu", is converted into a non-text image.

[0139] The graphic-text interaction method provided by the present disclosure is further illustrated below with a third application example.

[0140] like Figure 6 As shown, the graphic-text interaction method provided by the third application example of the present disclosure includes the following steps:

[0141] Step 601, obtaining a target image, wherein the target image is a non-text image;

[0142] Step 602, obtaining text information corresponding to the non-text image;

[0143] Step 603, obtaining a composition instruction;

[0144] Step 604: In response to the composition instruction, convert the target text information into composition text.

[0145] Figure 7 FIG. 7 is a schematic diagram of a graphic-text interaction device 700 provided in an embodiment of the present disclosure. Figure 7 As shown, the graphic-text interaction device 700 includes:

[0146] A first acquisition unit 701, used to acquire a target image;

[0147] A second acquisition unit 702 is used to acquire text information corresponding to the target image based on the label information of the target image; the label information is used to indicate whether the category of the target image is a text image or a non-text image;

[0148] The third acquisition unit 703 is used to acquire a first interaction instruction, where the first interaction instruction is used to indicate a user's demand for graphic-text interaction with the target image;

[0149] The fourth acquisition unit 704 is configured to acquire, in response to the first interaction instruction, a text-image interaction result of the target image based on the text information.

[0150] In one embodiment, the second obtaining unit 702 is specifically configured to:

[0151] Based on the label information of the target image, obtaining the category of the target image;

[0152] If the target image is a text image, obtaining the text contained in the text image;

[0153] If the target image is a non-text image, the text information corresponding to the non-text image is obtained.

[0154] In one embodiment, the second obtaining unit 702 is specifically configured to:

[0155] Acquire text position information in the text image;

[0156] Based on the text position information in the text image, the text in the text image is acquired.

[0157] In one embodiment, the second obtaining unit 702 is specifically configured to:

[0158] Obtaining text paragraph position information in the text image;

[0159] Segmenting the text paragraph position information to obtain text line position information in the text image;

[0160] The text line position information is segmented to obtain the single-word text position information in the text image.

[0161] In one embodiment, the second obtaining unit 702 is specifically configured to:

[0162] Based on the single-word text position information in the text image, identifying each single-word text in the text image;

[0163] Based on each single-word text in the text image and its corresponding single-word text position information, the text in the text image is acquired.

[0164] In one embodiment, the first interaction instruction includes a reading instruction, and the fourth acquisition unit 704 is specifically configured to:

[0165] Based on the reading instruction, obtaining target text information corresponding to the reading instruction;

[0166] In response to the reading instruction, the target text information is converted into audio information.

[0167] In one embodiment, the first interaction instruction includes a translation instruction, and the fourth acquisition unit 704 is specifically configured to:

[0168] Based on the translation instruction, obtaining target text information corresponding to the translation instruction;

[0169] In response to the translation instruction, the target text information is converted into first translated text information.

[0170] In one embodiment, the first interaction instruction includes a composition instruction, and the fourth acquisition unit 704 is specifically configured to:

[0171] Based on the composition instruction, obtaining target text information corresponding to the composition instruction;

[0172] In response to the composition instruction, the target text information is converted into image information.

[0173] In one embodiment, the first interaction instruction includes a composition instruction, and the fourth acquisition unit 704 is specifically configured to:

[0174] Based on the composition instruction, the text information is converted into composition text.

[0175] In one embodiment, the fourth obtaining unit 704 is further specifically configured to:

[0176] Obtaining a second interaction instruction;

[0177] In response to the second interactive instruction, the composition text is converted into second translation text information.

[0178] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.

[0179] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0180] Figure 8 8 is a block diagram of an electronic device 800 for implementing the above-mentioned image interaction method according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0181] Reference Figure 8, the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0182] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0183] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0184] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0185] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0186] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0187] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0188] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0189] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0190] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0191] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by the processor 820 of the electronic device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0192] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image interaction method described in the above embodiments of the present disclosure.

[0193] The embodiment of the present disclosure also provides a chip, which can be seen in Fig. 9 Schematic diagram of the chip structure shown. Fig. 9 The chip shown includes a processor 901 and an interface 902. The number of the processor 901 may be one or more, and the number of the interface 902 may be multiple.

[0194] Optionally, the chip further includes a memory 903, and the memory 903 is used to store necessary computer programs and data.

[0195] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0196] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0197] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0198] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0199] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0200] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0201] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0202] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for image-text interaction, characterized in that: include: Get the target image; Based on the label information of the target image, obtaining text information corresponding to the target image; The label information is used to indicate whether the category of the target image is a text image or a non-text image; Obtaining a first interactive instruction, wherein the first interactive instruction includes at least one of a reading instruction, a translation instruction, a composition instruction, or a text composition instruction; In response to the first interaction instruction, a graphic-text interaction result of the target image is obtained based on the text information.

2. The method according to claim 1, characterized in that The acquiring text information corresponding to the target image based on the label information of the target image includes: Based on the label information of the target image, obtaining the category of the target image; If the target image is a text image, obtaining the text contained in the text image; If the target image is a non-text image, the text information corresponding to the non-text image is obtained.

3. The method according to claim 2, characterized in that The obtaining of the text contained in the text image comprises: Acquire text position information in the text image; Based on the text position information in the text image, the text in the text image is acquired.

4. The method according to claim 3, characterized in that The obtaining of text position information in the text image includes: Obtaining text paragraph position information in the text image; Segmenting the text paragraph position information to obtain text line position information in the text image; The text line position information is segmented to obtain the single-word text position information in the text image.

5. The method according to claim 4, characterized in that The acquiring the text in the text image based on the text position information includes: Based on the single-word text position information in the text image, identifying each single-word text in the text image; Based on each single-word text in the text image and its corresponding single-word text position information, the text in the text image is acquired.

6. The method according to any one of claims 1 to 5, characterized in that The first interaction instruction includes a reading instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes: Based on the reading instruction, obtaining target text information corresponding to the reading instruction; In response to the reading instruction, the target text information is converted into audio information.

7. The method according to any one of claims 1 to 5, characterized in that The first interaction instruction includes a translation instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes: Based on the translation instruction, obtaining target text information corresponding to the translation instruction; In response to the translation instruction, the target text information is converted into first translated text information.

8. The method according to any one of claims 1 to 5, characterized in that The first interaction instruction includes a composition instruction, and the step of obtaining a text-image interaction result of the target image based on the text information in response to the first interaction instruction includes: Based on the composition instruction, obtaining target text information corresponding to the composition instruction; In response to the composition instruction, the target text information is converted into image information.

9. The method according to any one of claims 1 to 5, characterized in that The first interaction instruction includes a text composition instruction, and the step of obtaining the image-text interaction result of the target image based on the text information in response to the first interaction instruction includes: Based on the composition instruction, the text information is converted into composition text.

10. The method according to claim 9, characterized in that The method further comprises: Obtaining a second interaction instruction; In response to the second interactive instruction, the composition text is converted into second translation text information.

11. A graphic device, characterized in that: include: A first acquisition unit, used for acquiring a target image; A second acquisition unit, configured to acquire text information corresponding to the target image based on the label information of the target image; The label information is used to indicate whether the category of the target image is a text image or a non-text image; A third acquisition unit is used to acquire a first interaction instruction, where the first interaction instruction is used to indicate a user's demand for graphic-text interaction with the target image; A fourth acquisition unit is used to acquire a text-image interaction result of the target image based on the text information in response to the first interaction instruction.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

14. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 10.

Images