Image processing method and device, electronic equipment and storage medium
By identifying text and visual objects in the image and generating corresponding text and visual description text, combined with large language model processing, the shortcomings of complex image analysis in the existing technology are solved, deep understanding and comprehensive processing are achieved, and analysis accuracy is improved. User experience.
Patent Information
- Application Number
- CN202411248089.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-29
AI Technical Summary
Existing image analysis technology is difficult to process mixed content in complex images, especially in the identification and analysis of elements such as text, charts, and code. It lacks deep understanding and comprehensive processing capabilities, resulting in insufficient analysis accuracy and low interactivity.
By identifying text and visual objects in the image, corresponding text and visual description text are generated, prompt text is obtained based on these text and visual objects, image processing is combined with large language models to generate deep understanding and comprehensive image processing results.
It improves the processing effect and accuracy of content analysis of complex visual objects in the image, can automatically generate content summary, key information extraction and trend analysis, and is suitable for a variety of professional fields, enhancing user experience and operation efficiency.
Smart Images

Figure CN119049065A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision, deep learning, large models, and generative model technology. More specifically, the present disclosure provides an image processing method, device, electronic device, storage medium, and computer program product. Background Art
[0002] Current image analysis technologies are usually limited to simple object recognition and basic text extraction, lacking in-depth understanding and comprehensive processing of complex elements in images. Especially when processing images containing mixed content (such as text, graphics, codes, etc.), current image analysis technologies are often unable to accurately identify and effectively analyze image content. Summary of the invention
[0003] The present disclosure provides an image processing method, an apparatus, an electronic device, a storage medium, and a computer program product.
[0004] According to a first aspect, an image processing method is provided, the method comprising: recognizing text in an image to obtain image recognition text; recognizing visual objects in the image to obtain visual description text of the image; obtaining prompt text for processing the image based on the visual objects; and generating an image processing result based on the image recognition text, the visual description text and the prompt text.
[0005] According to a second aspect, an image processing device is provided, which includes: a first recognition module, used to recognize text in an image and obtain image recognition text; a second recognition module, used to recognize visual objects in the image and obtain visual description text of the image; an acquisition module, used to acquire prompt text for processing the image based on the visual objects; and a processing module, used to generate an image processing result based on the image recognition text, the visual description text and the prompt text.
[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.
[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.
[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.
[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which an image processing method and apparatus can be applied according to an embodiment of the present disclosure;
[0012] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure;
[0013] Figure 3 is a flowchart of an image processing method according to another embodiment of the present disclosure;
[0014] Figure 4 is a schematic diagram of an image processing effect according to an embodiment of the present disclosure;
[0015] Figure 5 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure;
[0016] Figure 6 is a block diagram of an electronic device according to an image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0018] Traditional image analysis technology mainly relies on basic OCR technology and simple image recognition algorithms. These methods mainly focus on the surface features and simple text content of images. For a variety of application fields that require advanced image processing functions, including image document management, content creation, online office, etc., especially in professional fields such as law, medical care, education and scientific research, traditional image analysis technology is difficult to meet the needs. It is mainly reflected in the following aspects.
[0019] Limited comprehension: Inability to process complex information in images, such as embedded tables, code, or specialized diagrams.
[0020] Insufficient analysis accuracy: When processing images within professional fields, it is often unable to provide sufficient analysis depth and accuracy.
[0021] Low interactivity: Lack of functionality to interact with users and inability to provide customized analysis results based on user needs.
[0022] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0023] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.
[0024] Figure 1 is a schematic diagram of an exemplary system architecture to which an image processing method and apparatus can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0025] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.
[0026] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.
[0027] The image processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can generally be set in the server 105. The image processing method provided in the embodiment of the present disclosure can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.
[0028] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure.
[0029] like Figure 2 As shown, the image processing method 200 includes operations S210 to S240.
[0030] In operation S210, text in an image is recognized to obtain image-recognized text.
[0031] The image may include text blocks and visual objects, and the visual objects may include code snippets, tables, charts, interface elements, etc. The text in the image may include plain text in the text block, and may also include text in visual objects such as code snippets, tables, charts, interface elements, etc.
[0032] Through OCR (Optical Character Recognition) technology, you can recognize the text in text blocks in images, and you can also recognize the text in visual objects such as code segments, tables, charts, and interface elements.
[0033] In operation S220 , a visual object in the image is identified to obtain a visual description text of the image.
[0034] For example, visual processing models and natural language processing models can be combined to identify visual objects in an image and generate visual description text for the image.
[0035] The visual processing model may include a detection model, a classification model, a segmentation model, an OCR recognition model, etc. The visual processing model may be used to obtain visual knowledge of each visual object in the image, such as location, category, text content, etc., and may also include attribute information such as color and shape. The natural language processing model may be a large language model. The above visual knowledge is input into the large language model, which may summarize, generalize, expand and modify the visual knowledge to obtain a visual description text.
[0036] For example, an image is input into a visual processing model to obtain the category, position, and text content of the visual objects in the image, and the above category, position, and text content are input into a large language model to obtain a visual description text. The visual description text can describe which visual objects are contained in the image, the category of each visual object (such as a code segment, a chart, etc.), and detailed description information of the visual object.
[0037] For example, the visual description text may describe: on the left side of the image is a code segment, the content of the code segment is XXX, on the right side of the image is a table, the content of each cell in the table is YYY, etc.
[0038] In operation S230, a prompt text for processing the image is acquired according to the visual object.
[0039] For different visual objects, corresponding prompt texts can be set. The prompt texts of different visual objects are used to perform special analysis on the corresponding visual objects to obtain corresponding analysis results.
[0040] For example, the prompt text for a code segment can be used to prompt the large model to parse out the function, usage method, and application scenario of the code segment.
[0041] The prompt text for the chart can be used to prompt the large model to parse out the key indicator information, trend information, etc. in the chart.
[0042] The prompt text for the table can be used to prompt large model generation trend analysis, etc.
[0043] The prompt text for the text block can be used to prompt the summary of the large model text, etc.
[0044] The prompt text for the interface elements can be used to prompt the large model to analyze the functions and effects of these interface elements.
[0045] In operation S240, an image processing result is generated according to the image recognition text, the visual description text, and the prompt text.
[0046] For example, the above-mentioned image recognition text, visual description text and prompt text can be input into a large model (such as a large language model). The large language model organizes the image recognition text and the visual description text to obtain the overall text. Based on the prompt text, the overall text is summarized, summarized, analyzed, etc. to obtain the image processing result.
[0047] The above-mentioned image processing results may include processing results of various visual objects in the image, for example, analysis results of code segments, charts, tables, etc.
[0048] The embodiments of the present disclosure recognize text in an image, obtain image recognition text, recognize visual objects in the image, obtain visual description text, obtain prompt text matching the visual objects, and generate an image processing result based on the image recognition text, the visual description text, and the prompt text. Since special prompt text can be selected for visual objects in the image, the processing effect of the visual objects in the image can be improved, and further, complex visual objects in the image, such as charts, codes, etc., can be deeply understood and comprehensively processed, thereby improving the accuracy of image content analysis.
[0049] Figure 3 is a flowchart of an image processing method according to another embodiment of the present disclosure.
[0050] like Figure 3 As shown, this embodiment includes operations S310 to S350.
[0051] In operation S310, text in an image is recognized to obtain image-recognized text.
[0052] According to an embodiment of the present disclosure, text in an image is recognized, and image-recognized text, as well as the position and attributes of the image-recognized text are obtained, wherein the attributes include at least one of a title, a chapter, and a paragraph.
[0053] For example, OCR technology can be used to perform text recognition on the text in the image to obtain image-recognized text. The OCR technology can also be optimized so that the position and attributes of the image-recognized text in the image can be identified. The position can include the coordinates of the two diagonal corners of the text block, and the attributes can include the title, chapter, and paragraph. Based on the attribute information such as title, chapter, and paragraph, the recognized text content can be organized into a standard format text for subsequent use and expansion.
[0054] The above OCR recognized text may include text in a separate text block, and may also include text in a visual object, such as code in a code segment, cell content in a table, etc.
[0055] The embodiments of the present disclosure can not only recognize standard printed text, but also optimize the recognition capabilities of paragraphs, chapters, titles, and text in complex backgrounds. Compared with traditional OCR technology, the OCR processing of the embodiments of the present disclosure is more accurate and more adaptable, and can process and obtain more abundant information, especially in the processing of non-standard text and office-related pictures. This enables the system to be used in a wider range of application scenarios, and the rich information obtained can be used for subsequent functional expansion.
[0056] In operation S320 , a visual object in the image is identified and a visual description text is generated.
[0057] According to an embodiment of the present disclosure, a visual object in an image is identified to obtain at least one of a position and a category of the visual object; and a visual description text of the image is generated based on at least one of the position and the category of the visual object and a sub-text in the visual object.
[0058] For example, visual processing models such as detection models, segmentation models, and classification models can be used to identify the position and category of each visual object in the image. The position can include the coordinates of the two diagonal corners of the bounding box of the visual object. The category can include code snippets, charts, tables, interface elements, etc.
[0059] In addition, visual objects may also contain text, such as the code in a code snippet, the content of each cell in a table, etc. Therefore, the OCR recognition model is also used as a visual processing model, and the recognized OCR text (i.e., sub-text) is also used as a visual processing result.
[0060] In addition, image processing models also include some entity recognition models, animal and plant recognition models, person recognition models, etc. These models can recognize attributes such as the names of people, buildings, animals and plants in the image.
[0061] The location, category, OCR text, and attribute information of the above-mentioned visual objects can be input into the large language model, and the large language model is expanded, supplemented, corrected and sorted to obtain the visual description text in the form of plain text description.
[0062] The embodiments of the present disclosure use OCR tools and various image processing tools based on visual processing models, combined with large models to intelligently identify various visual elements in images, and can distinguish complex image content such as tables, charts, code snippets and various interface elements.
[0063] The intelligent object recognition of the embodiments of the present disclosure goes beyond the traditional image classification function and can perform more detailed object classification and recognition, effectively improving the accuracy of recognition and the breadth of application. This makes information extraction from data tables, legal documents, medical reports, and technical manuals automatic and efficient, greatly reducing the burden of manual processing.
[0064] In operation S330 , prompt text is intelligently recommended according to the visual object category.
[0065] For different visual objects, the most suitable prompt text can be recommended.
[0066] According to an embodiment of the present disclosure, in response to the visual object being a code segment, prompt text for generating processing suggestions for the code segment is obtained, wherein the processing suggestions include at least one of function information, usage information, and application scenario information. In response to the visual object being a table or a chart, prompt text for generating a parsing result of the table or chart is obtained, wherein the parsing result includes at least one of content summary information, content extraction information, and trend analysis information. In response to the visual object containing a preset language text, prompt text for generating instruction recommendation information is obtained, wherein the instruction recommendation information includes instruction information for translating the preset language text.
[0067] For example, if the visual object in the image is a code segment, the recommended prompt text may include: This is a code segment, the code content is XXX, please explain in detail the function, usage and application scenario of this code segment.
[0068] For example, if the visual object in the image is a table, the recommended prompt text may include: This is a table, the table information includes YYY, please summarize the contents of the table, and generate a trend chart based on the table contents.
[0069] For example, if the visual object in the image is a chart, the recommended prompt text may include: This is a chart, the content of the chart includes ZZZ, please extract the key indicator information in the chart, etc.
[0070] For example, if the visual object in the image is a text block containing a preset language (such as English), the recommended prompt text may include: Please convert the English description into Chinese. This embodiment may also support conversion between multiple languages.
[0071] When an image contains visual objects of multiple categories, you can obtain the prompt text of each visual object separately.
[0072] The above prompt text can be pre-set, and the most suitable prompt text can be obtained according to the category of the visual object contained in the image, which can improve the accuracy of image parsing. Compared with traditional image analysis tools, the embodiments of the present disclosure can provide deeper semantic analysis and more comprehensive understanding of complex text and image content. It can automatically generate content summaries, key information extraction and trend analysis, which is particularly suitable for data-driven decision support.
[0073] In operation S340, the image recognition text, the visual description text, and the prompt text are input into the macro model.
[0074] In operation S350 , the large model is processed to obtain at least one of a parsing result, a processing suggestion result, and an instruction recommendation result.
[0075] According to an embodiment of the present disclosure, the image recognition text and the visual description text are integrated to obtain the integrated text; and at least one of the image analysis result, the processing suggestion result and the instruction recommendation result is generated based on the prompt text and the integrated text.
[0076] For example, when an image includes multiple visual objects, the image recognition text associated with each visual subject can be determined from the image recognition text based on location information, or keywords in the image recognition text, or the semantics of the image recognition text.
[0077] For each visual object, the OCR text, visual description text, and image recognition text associated with the visual object can be integrated to obtain the integrated text of the visual object. The integrated text and the prompt text matching the object are then input into a large model (such as a large language model) to obtain the parsing result of the visual object. The parsing results of multiple visual objects in the image are integrated to obtain the processing result of the image.
[0078] The above analysis results based on the big model not only display the analysis data (such as content summary, key information extraction and trend analysis, etc.), but also provide targeted improvement suggestions and optimization solutions, such as processing suggestions (such as functions, application scenarios, usage methods, etc.) and instruction recommendations (such as translation instructions).
[0079] The embodiments of the present disclosure are not only passive data presentations, but also active strategies and improvement measures. Users can obtain operational suggestions directly from the analysis results, such as pointing out potential growth areas in market analysis reports or proposing improvement measures in technical documents, etc., which enhances practicality, simplifies operational steps, improves operational efficiency, and further improves user experience.
[0080] Figure 4 is a schematic diagram of an image processing effect according to an embodiment of the present disclosure.
[0081] like Figure 4 As shown, the image 410 includes a code segment, a table and a text block. After the image 410 is processed by the image processing method provided by the embodiment of the present disclosure, an image processing result 420 is obtained.
[0082] For example, text recognition can be performed on image 410 to obtain image recognition text. The image recognition text includes the code in the code segment, the text in the table, and the text in the text block. Next, the visual objects in image 410 are identified, and the categories of the visual objects, such as code segments, tables, and text blocks. The position information of each visual object is also identified. The category, position, and text contained in each visual object can be input into the large language model. The visual description text of each visual object is obtained.
[0083] Next, you can get the prompt text for the code segment, such as "Please explain the function, usage and application scenario of this code in detail." Get the prompt text for the table, such as "Please summarize the content in the table and analyze the trend of each indicator." Get the prompt text for the text block, such as "Generate a text summary."
[0084] Next, for each visual object, the image recognition text associated with the visual object, the text of the visual object itself, the visual description text and the prompt text can be input into the large language model, and the large language model will organize, expand, supplement and correct them to obtain the image processing result 420.
[0085] For example, the image processing result 420 includes the specific analysis result of the code segment "On the left side of the image is a segment of code used to generate a front-end interactive page...", and also includes the specific analysis result of the table "There is a table in the upper right corner of the image, and indicator A in the table shows an increasing trend..." It also includes the specific analysis result of the text block "The text summary in the lower right corner of the image is...".
[0086] The image processing results generated by the embodiments of the present disclosure not only display the analysis data, but also provide targeted improvement suggestions and optimization solutions. It is not just a passive data display, but an active strategy and improvement measure, which enhances practicality, simplifies operation steps, improves operation efficiency, and further improves user experience.
[0087] According to an embodiment of the present disclosure, the present disclosure also provides an image processing device.
[0088] Figure 5 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.
[0089] like Figure 5 As shown, the image processing device 500 includes a first recognition module 510 , a second recognition module 520 , an acquisition module 530 and a processing module 540 .
[0090] The first recognition module 510 is used to recognize text in an image and obtain image-recognized text.
[0091] The second recognition module 520 is used to recognize visual objects in the image and obtain visual description text of the image.
[0092] The acquisition module 530 is used to acquire prompt text for processing the image according to the visual object.
[0093] The processing module 540 is used to generate an image processing result according to the image recognition text, the visual description text and the prompt text.
[0094] The processing module 540 includes an integrating unit and a generating unit.
[0095] The integration unit is used to integrate the image recognition text and the visual description text to obtain an integrated text.
[0096] The generating unit is used to generate at least one of an image analysis result, a processing suggestion result and an instruction recommendation result according to the prompt text and the integrated text.
[0097] The acquisition module 530 includes a first acquisition unit, a second acquisition unit and a third acquisition unit.
[0098] The first acquisition unit is used to acquire a prompt text for generating a processing suggestion for the code segment in response to the visual object being a code segment, wherein the processing suggestion includes at least one of function information, usage information, and application scenario information.
[0099] The second acquisition unit is used to acquire prompt text of a parsing result for generating a table or a chart in response to the visual object being a table or a chart, wherein the parsing result includes at least one of content summary information, content extraction information, and trend analysis information.
[0100] The third acquisition unit is used to acquire a prompt text for generating instruction recommendation information in response to the visual object containing a preset language text, wherein the instruction recommendation information includes instruction information for translating the preset language text.
[0101] The first recognition module 510 is used to recognize text in an image, obtain image-recognized text, and the position and attributes of the image-recognized text, where the attributes include at least one of a title, a chapter, and a paragraph.
[0102] The text recognized by the first recognition module 510 includes subtext in the visual object. The second recognition module 520 includes a recognition unit and a generation unit.
[0103] The recognition unit is used to recognize the visual object in the image and obtain at least one of the position and category of the visual object.
[0104] The generating unit is used to generate a visual description text of the image according to at least one of a position and a category of the visual object and a subtext in the visual object.
[0105] According to an embodiment of the present disclosure, the visual object includes at least one of a code segment, a table, a chart, a text block, and an interface element.
[0106] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0107] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0108] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0109] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0110] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).
[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0112] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0116] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. An image processing method, comprising: Recognize the text in the image and obtain the image recognition text; Identify visual objects in the image and obtain visual description text of the image; According to the visual object, obtaining a prompt text for processing the image; as well as An image processing result is generated according to the image recognition text, the visual description text and the prompt text.
2. The method according to claim 1, wherein: The generating of the image processing result according to the image recognition text, the visual description text and the prompt text comprises: integrating the image recognition text and the visual description text to obtain an integrated text; and At least one of an analysis result, a processing suggestion result, and an instruction recommendation result of the image is generated according to the prompt text and the integrated text.
3. The method according to claim 1, wherein: The acquiring, according to the visual object, a prompt text for processing the image comprises: In response to the visual object being a code segment, a prompt text for generating a processing suggestion for the code segment is obtained, wherein the processing suggestion includes at least one of function information, usage information, and application scenario information.
4. The method according to claim 1, wherein: The acquiring, according to the visual object, a prompt text for processing the image comprises: In response to the visual object being a table or a chart, a prompt text of a parsing result for generating the table or the chart is obtained, wherein the parsing result includes at least one of content summary information, content extraction information, and trend analysis information.
5. The method according to claim 1, wherein: The acquiring, according to the visual object, a prompt text for processing the image comprises: In response to the visual object including text in a preset language, a prompt text for generating instruction recommendation information is obtained, wherein the instruction recommendation information includes instruction information for translating the text in the preset language.
6. The method according to claim 1, wherein: Recognizing the text in the image to obtain the image recognized text includes: The text in the image is recognized to obtain the image-recognized text, and the position and attributes of the image-recognized text, wherein the attributes include at least one of a title, a chapter, and a paragraph.
7. The method according to claim 6, wherein: The text includes subtext in the visual object; The step of identifying the visual object in the image and obtaining a visual description text of the image comprises: Identify a visual object in the image and obtain at least one of a position and a category of the visual object; as well as A visual description text of the image is generated according to at least one of a position and a category of the visual object and a subtext in the visual object.
8. The method according to claim 1, wherein: The visual object includes at least one of a code segment, a table, a chart, a text block, and an interface element.
9. An image processing device, comprising: The first recognition module is used to recognize text in the image and obtain image recognition text; A second recognition module is used to recognize visual objects in the image and obtain visual description text of the image; An acquisition module, used for acquiring a prompt text for processing the image according to the visual object; as well as A processing module is used to generate an image processing result according to the image recognition text, the visual description text and the prompt text.
10. The device according to claim 9, wherein: The processing module comprises: an integration unit, used for integrating the image recognition text and the visual description text to obtain an integrated text; and A generating unit is used to generate at least one of an analysis result, a processing suggestion result and an instruction recommendation result of the image according to the prompt text and the integrated text.
11. The device according to claim 9, wherein: The acquisition module comprises: The first acquisition unit is used to acquire a prompt text for generating a processing suggestion for the code segment in response to the visual object being a code segment, wherein the processing suggestion includes at least one of function information, usage information and application scenario information.
12. The device according to claim 9, wherein: The acquisition module comprises: The second acquisition unit is used to acquire prompt text of the analysis result used to generate the table or chart in response to the visual object being a table or chart, wherein the analysis result includes at least one of content summary information, content extraction information, and trend analysis information.
13. The device according to claim 9, wherein: The acquisition module comprises: The third acquisition unit is used to acquire a prompt text for generating instruction recommendation information in response to the visual object containing a preset language text, wherein the instruction recommendation information includes instruction information for translating the preset language text.
14. The device according to claim 9, wherein: The first recognition module is used to recognize text in an image, obtain image-recognized text, and the position and attributes of the image-recognized text, wherein the attributes include at least one of a title, a chapter, and a paragraph.
15. The device according to claim 14, wherein: The text includes subtext in the visual object; The second identification module comprises: a recognition unit, configured to recognize a visual object in the image and obtain at least one of a position and a category of the visual object; as well as A generating unit is used to generate a visual description text of the image according to at least one of a position and a category of the visual object and a subtext in the visual object.
16. The device according to claim 9, wherein: The visual object includes at least one of a code segment, a table, a chart, a text block, and an interface element.
17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.
18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.
19. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
