Image processing method and device, equipment and storage medium
By identifying and correlating text information in images and translating in specific fields such as medical fields, the problems of inaccurate translation, poor interactivity, inconsistent typesetting and low statement correlation in the prior art are solved, and efficient and accurate image translation effect is achieved.
Patent Information
- Application Number
- CN202411773810.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-03-28
AI Technical Summary
The existing image translation solutions have problems such as the medical field and other specific fields such as the medical field, such as the problem of inaccurate translation, poor interactivity, inconsistent typesetting and low statement correlation.
By identifying multiple target areas in the image to be processed, the text information is recognized using the trained recognition model, and after correlation is performed based on the degree of relevance of the text information, the text information is translated into another language, and the original text information is replaced in the image to achieve translation.
It realizes accurate identification and correlation of text information in any shape, ensures that the translated text paragraphs are clear and the sentences are smooth, and closely combines the translated text with the images, maintains the original layout method, and improves the translation effect.
Smart Images
Figure CN119723584A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, device and storage medium for image processing. Background Art
[0002] Image translation refers to converting the text or symbol content in an image into another language or expression. Usually, the image translation solution uses optical character recognition (OCR) technology to recognize the text or symbol content in the image and convert it into editable text, thereby converting the text or symbol content into another language or expression. However, this image translation solution is not applicable to certain specific fields (for example, the medical field). Therefore, there is an urgent need for an image translation solution that can be applied to certain specific fields. Summary of the invention
[0003] In a first aspect of the present disclosure, a method for image processing is provided. The method includes: in response to receiving an image to be processed, identifying multiple target areas in the image to be processed, wherein the multiple target areas at least completely cover corresponding text information in the image to be processed; identifying corresponding text information in the multiple target areas through a trained recognition model; associating the corresponding text information based on the correlation between two adjacent text information in the corresponding text information; translating the associated corresponding text information from one language into another language to obtain corresponding translated text information; and replacing the corresponding text information with the corresponding translated text information in the image to be processed to obtain a target image.
[0004] In a second aspect of the present disclosure, a device for image processing is provided. The device includes: a first recognition module, configured to identify multiple target areas in the image to be processed in response to receiving the image to be processed, and the multiple target areas at least completely cover the corresponding text information in the image to be processed; a second recognition module, configured to identify the corresponding text information in the multiple target areas through a trained recognition model; an association module, configured to associate the corresponding text information based on the association degree of two adjacent text information in the corresponding text information; a translation module, configured to translate the associated corresponding text information from one language to another language to obtain corresponding translated text information; and a replacement module, configured to replace the corresponding text information with the corresponding translated text information in the image to be processed to obtain a target image.
[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect of the present disclosure.
[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to perform the method according to the first aspect of the present disclosure.
[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, wherein the computer executable instructions implement the method according to the first aspect of the present disclosure when executed by a processor.
[0008] It should be understood that the contents described in the summary of the present invention are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;
[0011] Figure 2 A flowchart showing a process for image processing according to some embodiments of the present disclosure is shown;
[0012] Figure 3a A schematic diagram showing a text area in an image according to some embodiments of the present disclosure;
[0013] Figure 3b A schematic diagram showing candidate regions in an image according to some embodiments of the present disclosure;
[0014] Figure 3c A schematic diagram showing a target area in an image according to some embodiments of the present disclosure;
[0015] Figure 4 A flowchart showing a process for target strategy determination according to some embodiments of the present disclosure is shown;
[0016] Figure 5A flowchart showing a process for image translation in a medical scenario according to some embodiments of the present disclosure is shown;
[0017] Figure 6a A schematic diagram showing a CT image for a user according to some embodiments of the present disclosure is shown;
[0018] Figure 6b A schematic diagram showing candidate regions in a CT image according to some embodiments of the present disclosure is shown;
[0019] Figure 6c A schematic diagram showing a target area in a CT image according to some embodiments of the present disclosure;
[0020] Figure 6d A schematic diagram showing a CT image after performing paragraph association according to some embodiments of the present disclosure;
[0021] Figure 6e A schematic diagram showing a CT image after text information is removed according to some embodiments of the present disclosure;
[0022] Figure 6f A schematic diagram showing a CT image after adding translated text information according to some embodiments of the present disclosure;
[0023] Figure 7a A schematic diagram showing a user's medical record according to some embodiments of the present disclosure is shown;
[0024] Figure 7b A schematic diagram showing an example of paragraph division in a user's medical record according to some embodiments of the present disclosure;
[0025] Figure 7c A schematic diagram showing translation results of a user's medical record using the image processing solution provided by the present disclosure according to some embodiments of the present disclosure is shown;
[0026] Figure 7d A schematic diagram showing translation results of a user's medical records in a traditional manner according to some embodiments of the present disclosure is shown;
[0027] Figure 8 A block diagram showing an apparatus for image processing according to some embodiments of the present disclosure; and
[0028] Fig. 9 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION
[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0030] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0031] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.
[0032] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, "model" may also be referred to as "machine learning model", "machine learning network" or "network", and these terms are used interchangeably in this article.
[0033] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association from input to output (also called the mapping of input to output) from the training data. The parameter values of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values obtained from the training to determine the corresponding output.
[0034] As briefly mentioned above, current image translation solutions are not suitable for certain specific fields. Taking the medical field as an example, as people's health awareness increases, the demand for convenient, professional and personalized medical services is growing. Especially in today's rapid development of artificial intelligence (AI) technology, intelligent medical solutions based on AI technology have emerged to meet people's diverse needs for health management.
[0035] However, despite the large user base of Internet medical applications, many users lack effective image translation tools in their own country when obtaining medical services through the Internet. For example, when diagnosing computed tomography (CT) images taken in other countries, doctors may not be able to understand the case information when processing these case materials, which is not conducive to the diagnosis of patients.
[0036] The current image translation scheme mainly uses OCR technology to recognize the text or symbol content in the image and convert it into editable text, thereby converting the text or symbol content into another language or expression. However, such image translation schemes have the following problems: the translation is not accurate and authoritative enough; the interactivity in the translation process is poor (for example, the translated text requires the user to manually select or directly output the translated text, and is not combined with the image); the layout of the translated text is inconsistent with the layout of the text in the original image; the correlation between sentences is poor, etc.
[0037] In order to at least partially solve the above problems and other potential problems that may exist in traditional solutions, the present disclosure provides a solution for image processing. According to various embodiments of the present disclosure, multiple target areas in the image to be processed are identified, and the multiple target areas at least completely cover the corresponding text information in the image to be processed. The corresponding text information in the multiple target areas is identified through a trained recognition model. Based on the correlation between two adjacent text information in the corresponding text information, the corresponding text information is associated. The associated corresponding text information is translated from one language into another language to obtain the corresponding translated text information. In the image to be processed, the corresponding text information is replaced with the corresponding translated text information to obtain a target image. In this way, text information of any shape can be identified, and the adjacent text information identified is associated to achieve paragraph correction, thereby ensuring that the translated text information has clear paragraphs and smooth sentences. At the same time, the translated text information and the image are combined to ensure that the typesetting method after translation is close to the original typesetting method.
[0038] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.
[0039] Figure 1 A schematic diagram of an example environment 100 in which an embodiment of the present disclosure can be implemented is shown. In the environment 100, an application 130 for image translation is deployed in a terminal device 110, such as a web application or other types of applications. In some embodiments, the application 130 is provided to assist a user 170 in various task processing requirements in different applications and scenarios (e.g., medical image translation). During the interaction process with the application 130, the user 170 inputs an interactive message, and the application 130 provides a reply message in response to the user input. In some embodiments, the interactive message with the application 130 may include messages in a multimodal form, such as text messages (e.g., natural language text), voice messages, image messages, video messages, and the like.
[0040] In the environment 100, a recognition model 140, an association model 150, and a translation model 160 are deployed in the server 120. In some embodiments, the recognition model 140 may be, for example, a model for processing image data or a model for processing video data.
[0041] Take the interactive information input by the user 170 as an example, which is an image message. The recognition model 140 can recognize the text information in the image message, the association model 150 can associate the recognized text information with paragraphs, the translation model 160 can translate the associated text information, and the server 120 returns the translated image message to the terminal device 110, which is presented to the user 170 through the application 130.
[0042] In some embodiments, the terminal device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDA), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the client device can also support any type of interface for the user (such as "wearable" circuits, etc.).
[0043] In some embodiments, the server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 120 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.
[0044] A communication connection may be established between the server 120 and the terminal device 110. The communication connection may be established in a wired manner or a wireless manner. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In the embodiments of the present disclosure, the server 120 and the terminal device 110 may implement signaling interaction through the communication connection between the two.
[0045] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure.
[0046] Figure 2 FIG. 2 is a flowchart of a process 200 for image processing according to some embodiments of the present disclosure. The process 200 may be implemented in the server 120. Figure 1 The process 200 is described in detail.
[0047] In block 210, the server 120 identifies a plurality of target regions in the image to be processed in response to receiving the image to be processed sent by the terminal device 110. The plurality of target regions at least completely cover the corresponding text information in the image to be processed.
[0048] In some embodiments, the image to be processed may be an image input by the user 170 to the terminal device 110 and then transmitted by the terminal device 110 or transmitted in real time to the server 120, or an image acquired by the terminal device 110 and transmitted or transmitted in real time to the server 120, or an image input by the user 170 or an image acquired by the terminal device 110 after preprocessing and then transmitted to the server 120. For example, the preprocessing of the image input by the user 170 or acquired by the terminal device 110 may be adjusting the resolution of the image, dividing the image into images of a predetermined size, and the like.
[0049] In some embodiments, the image to be processed may also be a frame in a video input by the user 170. For example, the terminal device 110 receives a video input by the user 170, and determines a frame from the video as the image to be processed based on the user's designation.
[0050] In some embodiments, the image to be processed may include one or more text information, and one text information may indicate a word, a sentence, or a paragraph of text. The outer edge of each text information may be determined based on the font size and typesetting of the text information. For example, the outer edge of each text information in the image to be processed may be a regular graphic or an irregular graphic. The embodiments of the present disclosure are not limited to this.
[0051] When identifying the target area in the image to be processed, the traditional OCR technology mainly recognizes and detects the text in the image at the line granularity. Under this premise, there will be the following problems: if the text of a single line is incomplete, the translation will be inaccurate, resulting in semantic obstruction; if the text in the entire image is translated, the text is too long, which will cause the interface to time out or the translated text content is difficult to split, and the layout is inconsistent with the original text content. Therefore, based on the OCR technology, the paragraph dimension method can be used to achieve the translation and integration of text in the image. OCR technology refers to the process in which an electronic device (such as a scanner or a digital camera) checks the characters printed on paper, determines its shape by detecting dark and light patterns, and then translates the shape into computer text using a character recognition method, that is, for printed characters, the text in the paper document is converted into a black and white dot matrix image file by optical means, and the text in the image is converted into a text format through recognition software for further editing and processing by word processing software. How to debug or use auxiliary information to improve the recognition accuracy is the most important topic of OCR, and the term Intelligent Character Recognition (ICR) technology is also generated for this reason. The main indicators for measuring the performance of an OCR system include: rejection rate, false recognition rate, recognition speed, user interface friendliness, product stability, ease of use and feasibility, etc.
[0052] Traditional paragraph detection mainly uses a segmentation-based text detection algorithm (Differentiable Binarization Network, DBNet). However, DBNet is suitable for regular text areas (for example, quadrilaterals). For irregular text areas, there will be errors in processing, and it is difficult to distinguish adjacent texts. To this end, in the disclosed embodiment, the server 120 can identify multiple target areas in the image to be processed. Specifically, first, the server 120 can identify multiple candidate areas in the image to be processed, and the multiple candidate areas at least cover part of the corresponding text information. Then the server 120 adjusts the identified multiple candidate areas so that the multiple candidate areas completely cover the corresponding text information, thereby obtaining multiple target areas.
[0053] In some embodiments, first, the server 120 can use multi-point coordinates to delineate the scope of the preliminary text information. In the embodiments of the present disclosure, the scope of the text information that is initially delineated is also referred to as a candidate area. Then, the server 120 predicts the scope of the text information with different shrinkage ratios and expands the scope of the detected text information one by one to obtain the target area. In the embodiments of the present disclosure, the above process of determining the target area is also referred to as progressive scale expansion to perform segmentation detection on the text area.
[0054] As an example, Figure 3a A schematic diagram of a text region 300A in an image 300 is shown according to some embodiments of the present disclosure. Figure 3b A schematic diagram of a candidate region 300B in an image 300 according to some embodiments of the present disclosure is shown. Figure 3c A schematic diagram of a target area 300C in an image 300 is shown according to some embodiments of the present disclosure.
[0055] like Figure 3a , Figure 3b and Figure 3c As shown, the image 300 to be processed includes three segments of text information 300A. The server 120 can preliminarily determine the candidate areas 300B corresponding to the three segments of text information 300A in the image 300 to be processed by means of multi-point coordinate delineation. Each candidate area 300B at least covers part of the corresponding text information 300A. By predicting different shrinkage ratios, the coverage of each candidate area 300B is expanded one by one, so that each candidate area 300B completely covers the corresponding text information, thereby obtaining multiple target areas 300C.
[0056] It should be understood that the above example only illustrates the case where the candidate area does not completely cover the text information. In the case where the candidate area completely covers the text information, it is necessary to reduce the coverage of the candidate area. It should be noted that when reducing the coverage of the candidate area, the candidate area can be reduced to the point where it does not completely cover the text information.
[0057] In this way, the detection and segmentation of text information of any shape and adjacent to each other can be realized. For example, when the outer contour of the text information in the image to be processed is irregular, the area where the text information is located can be accurately determined.
[0058] Continue to see Figure 2 In box 220 , the server 120 recognizes corresponding text information in multiple target areas through the trained recognition model 140 .
[0059] In some embodiments, the recognition model 140 can be a neural network model of any suitable type. For example, the recognition model 140 can be a bidirectional encoder representation based on Transformer (Bidirectional Encoder Representations from Transformers, BERT). In some embodiments, the recognition model 140 is trained based on data in a predetermined field. It should be understood that those skilled in the art can select the field type of data used to train the recognition model according to actual needs, and the embodiments of the present disclosure are not limited thereto.
[0060] Taking the medical field as an example, the data used to train the recognition model 140 may be medical information, such as but not limited to: diagnosis, prescription, consultation record, electronic medical record, consultation summary, etc.
[0061] In block 230 , the server 120 associates the corresponding text information based on the association degree of two adjacent text information in the corresponding text information.
[0062] The server 120 can input the identified multiple text information into the association model 150, determine the association degree between two adjacent text information in the multiple text information through the association model 150, and associate the corresponding text information based on the association degree. It should be understood that the association degree is used to reflect whether there is a correlation between two adjacent text information, so as to determine whether the two adjacent text information belong to the same sentence or the same paragraph. In some embodiments, a text information can include multiple sub-text information. The server 120 can also determine the association degree between two adjacent sub-text information in the multiple sub-text information through the association model 150, and associate the multiple sub-text information based on the association degree.
[0063] In some embodiments, each target area may include multiple text information. Two adjacent text information may indicate two adjacent text information located in the same target area, or may indicate two adjacent text information located in different target areas. Regardless of whether the two adjacent text information are located in the same target area, the two adjacent text information may be associated based on the degree of association, thereby avoiding the existence of adjacent text information in two adjacent target areas due to unreasonable target area division, thereby facilitating subsequent processing and making the translation result more accurate.
[0064] In some embodiments, in response to the correlation exceeding a correlation threshold, the server 120 may associate two adjacent text information, that is, treat the two adjacent text information as the same sentence or the same paragraph.
[0065] In some embodiments, the server 120 may determine the target words in two adjacent text messages respectively through the association model 150, and determine the ratio of the number of simultaneous occurrences of the target words in the two adjacent text messages to the total number of occurrences of the target words in the two adjacent text messages as the association degree.
[0066] In some embodiments, the association model 150 can be trained using data from a predetermined field. For example, in the medical field, the association model 150 can be trained using medical data. In some embodiments, the association model 150 mainly optimizes the crossover operator to determine whether two adjacent text information are related. Specifically, for two text individuals, for example, X and Y, the correlation between X and Y can be calculated by the following formula:
[0067]
[0068] Among them, s is the correlation between two text individuals, c is the number of times the two texts appear simultaneously in the model, and n is the number of times text X or text Y is included in the model.
[0069] Based on the calculation results of the above formula, the larger the value of s is, the greater the correlation between X and Y is, that is, the closer the number of times X and Y appear at the same time c is to n, that is, the greater the probability of X and Y appearing at the same time.
[0070] In some embodiments, the value of 1-s may be used as the elimination rate, as shown in the following formula:
[0071]
[0072] Among them, P c The elimination rate.
[0073] Combining the above two formulas, we can see that the lower the correlation s is, the higher the elimination rate P is. cThe higher the correlation s is, the lower the probability that two text individuals belong to the same paragraph. At the same time, the higher the correlation s is, the higher the elimination rate P is. c The lower it is, the higher the probability that the two text individuals belong to the same paragraph. In this way, multiple text information can be associated. For example, if the correlation between two adjacent text information in multiple text information is high, the two adjacent text information can be associated; otherwise, the two adjacent text information are not associated. For example, if the elimination rate of one text information is lower than that of the other text information in two adjacent text information, the two adjacent text information can be associated; otherwise, the two adjacent text information are not associated.
[0074] In block 240 , the server 120 translates the associated corresponding text information from one language into another language to obtain corresponding translated text information.
[0075] The server 120 can use the associated corresponding text information as input to the translation model 160, thereby obtaining corresponding translated text information. In some embodiments, the translation model 160 can be trained using data from a predetermined field. For example, in the medical field, the translation model 160 can be trained using medical data.
[0076] In some embodiments, the server 120 may input the translated text information into a trained compliance judgment model to determine the compliance of the translated text information. In some embodiments, the compliance judgment model may also be trained using data from the same field as the translation model 160 .
[0077] In block 250 , the server 120 replaces the corresponding text information with the corresponding translated text information in the image to be processed to obtain a target image.
[0078] In some embodiments, the server 120 may remove the corresponding text information from the image to be processed, fill the multiple target areas with colors that coordinate with the surrounding images, and then add the corresponding translated text information to the image to be processed.
[0079] Traditional image text erasure and backfilling techniques usually use text segmentation methods and introduce various loss functions to improve performance, which may lead to difficulties in selecting appropriate hyperparameters. In order to reduce this difficulty, this paper will erase the text in the image in the following way.
[0080] In some embodiments, the server 120 removes the corresponding text information from the image to be processed and fills the multiple target areas with colors that are coordinated with the surrounding images based on the target strategy. Figure 4 This paper introduces the process of determining the target strategy. Figure 4A flow chart of a process 400 for target policy determination according to some embodiments of the present disclosure is shown. The process 400 may be implemented in the server 120.
[0081] In box 410, the server 120 determines a first image and a second image based on the image to be processed. The first image is an image obtained by removing multiple target areas in the image to be processed. The second image is an image obtained by filling multiple target areas with colors that are coordinated with the surrounding images. In some embodiments, the color that is coordinated with the surrounding images can be any suitable color, as long as it is ensured to be coordinated with the colors around each target area. In this article, coordination can be understood as the same or similar colors. Alternatively or additionally, multiple target areas can be filled with the background color of the image.
[0082] In block 420, the server 120 determines, in a predetermined feature space, a first feature representation of the first image, a second feature representation of the second image, and a third feature representation of the plurality of target regions. In some embodiments, the predetermined feature space may be any suitable feature space, and the present disclosure is not limited thereto. For example, in the predetermined feature space, the first feature representation is f(x p ), the second feature is expressed as f(x n ), the third feature is expressed as f(x a ).
[0083] In block 430, the server 120 determines a first distance between the first feature representation and the third feature representation. From the first feature representation and the third feature representation in the predetermined feature space, it can be seen that the first distance can be expressed as |f(x a )-f(x p )|.
[0084] In block 440, the server 120 determines a second distance between the second feature representation and the third feature representation. From the second feature representation and the third feature representation in the predetermined feature space, it can be seen that the second distance can be expressed as |f(x a )-f(x n )|.
[0085] At block 450, the server 120 determines a target strategy based on the first distance, the second distance, and a predetermined threshold. In some embodiments, the server 120 may represent the first feature as a positive direction, the second feature as a negative direction, and the third feature as an anchor point. In the predetermined feature space, there may be the following inequality:
[0086] |f(x a )-f(x p )|-α<|f(x a )-f(x n )|
[0087] Among them, α is the parameter to be adjusted.
[0088] Based on the above formula, the negative distance (i.e., the second distance) in the predetermined feature space can be forced to be closer and the positive distance (i.e., the first distance) to be farther. That is to say, when clearing the anchor points, it is necessary to clear the text first and then cover the area with a color that is coordinated with the colors around each target area.
[0089] Thus, the following loss function can be obtained:
[0090] L=[|f(x a )-f(x p )|-|f(x a )-f(x n )|-α]
[0091] When removing text information from the image to be processed, the optimal solution of L can be determined by adjusting the value of α. For example, the value of α can be adjusted to increase L. For example, when α is close to 0, L reaches the optimal solution. It should be noted that α close to 0 can be understood as the difference between α and 0 is less than 0.001.
[0092] In this way, the server 120 can determine the optimal target strategy, remove the corresponding text information from the image to be processed based on the target strategy, and fill the multiple target areas with colors that coordinate with the surrounding images.
[0093] In some embodiments, the server 120 adds the corresponding translated text information to the image to be processed, and any method in the related art may be used, which is not limited in the embodiments of the present disclosure. However, the target image obtained after adding the corresponding translated text to the image to be processed needs to meet a predetermined condition.
[0094] In some embodiments, the predetermined condition may include at least one of the following: the position of the corresponding translated text information in the target image is the same as the position of the corresponding text information in the image to be processed, the layout of the corresponding translated text information is the same as the layout of the corresponding text information, or the clarity of the target image meets the predetermined clarity.
[0095] In some embodiments, the server 120 may obtain the typesetting of the corresponding text information, and further adjust the corresponding translated text information based on the typesetting, so that the adjusted typesetting of the corresponding translated text information is the same as the typesetting of the corresponding text information. Finally, the adjusted corresponding translated text information is added to the image to be processed.
[0096] In some embodiments, in the process of adding the corresponding translated text information to the image to be processed, the server 120 may also determine the position information of multiple target areas in the image to be processed. For example, the image to be processed is position-coded to determine the position information of multiple target areas in the image to be processed. After determining the position information of multiple target areas, the server 120 may add the corresponding translated text information to the image to be processed based on the position information, so that the position of the corresponding translated text information in the target image is the same as the position of the corresponding text information in the image to be processed.
[0097] In some embodiments, the server 120 may further adjust the definition of the target image so that the definition of the target image meets a predetermined definition (eg, is higher than a predetermined definition).
[0098] In order to more clearly explain the solutions for image processing provided by some embodiments of the present disclosure, the following description is made in conjunction with an example of image translation in a medical scenario. Of course, it is understandable that the embodiments of the present disclosure are not limited to this application scenario.
[0099] Figure 5 FIG. 5 is a flowchart of a process 500 for image translation in a medical scenario according to some embodiments of the present disclosure. The process 500 may be implemented in the server 120. Figure 1 Process 500 is introduced.
[0100] In block 510 , the server 120 receives an image transmitted by the terminal device 110 .
[0101] For example, the user 170 may send a computed tomography (CT) image for the user 170 to the terminal device 110 via the application 130, and the terminal device 110, in response to receiving the CT image, sends the CT image to the server 120. As an example, see Figure 6a , Figure 6a A schematic diagram of a CT image 600A for a user according to some embodiments of the present disclosure is shown.
[0102] Continue to see Figure 5 In block 520, the server 120 performs paragraph segmentation on the received image. In some embodiments, the server 120 may first identify candidate regions of the received image that cover text information. Then, the candidate regions are adjusted to obtain a target region. Finally, the text information in the target region is identified to associate the identified text information with paragraphs. As an example, see Figure 6b to Figure 6d , Figure 6b FIG. 6 is a schematic diagram showing candidate regions in a CT image 600B according to some embodiments of the present disclosure. Figure 6cFIG. 6 is a schematic diagram showing a target area in a CT image 600C according to some embodiments of the present disclosure. Figure 6d A schematic diagram of a CT image 600D after performing paragraph association according to some embodiments of the present disclosure is shown.
[0103] In some embodiments, during the process of segmenting the image by paragraphs, the server 120 may associate the text information in the image by paragraphs to determine whether two adjacent text information belong to the same paragraph. After association, the server 120 may translate the associated multiple text information. In some embodiments, the associated multiple text information may be translated by a stop translation method to obtain translated text information, which may improve the efficiency of translation and solve the problem of translation timeout.
[0104] Continue to see Figure 5 In block 530 , the server 120 performs paragraph association and image-text correction on the translated text information.
[0105] In some embodiments, when receiving an image sent by the terminal device 110 , the server 120 may perform position encoding on multiple text information in the image to determine the position information of each text information.
[0106] In the paragraph association and image-text correction stage, the server 120 can remove multiple text information in the image and fill the area where the multiple text information is removed with the background color of the image. Then, based on the position information of the multiple text information, the corresponding translated text information is added to the corresponding position. As an example, see Figure 6e and Figure 6f , Figure 6e FIG. 6 is a schematic diagram showing a CT image 600E after text information is removed according to some embodiments of the present disclosure. Figure 6f A schematic diagram of a CT image 600F after adding translated text information according to some embodiments of the present disclosure is shown.
[0107] The following is a comparison to illustrate the difference between the image processing solution provided by the embodiment of the present disclosure and the traditional solution. Figures 7a to 7d , Figure 7a FIG. 7 is a schematic diagram showing a user medical record 700A according to some embodiments of the present disclosure. Figure 7b A schematic diagram showing an example of paragraph division in a user medical record 700B according to some embodiments of the present disclosure, Figure 7c A schematic diagram showing a translation result 700C of a user's medical record using the image processing solution provided by the present disclosure according to some embodiments of the present disclosure is shown. Figure 7d A schematic diagram showing a translation result 700D for a user's medical record using a traditional method according to some embodiments of the present disclosure is shown.
[0108] In user medical record 700A, four text areas are framed by boxes, and the text information in the four text areas can be processed asynchronously, thereby improving the efficiency of translation. In user medical record 700B, the two lines of text in area b-1 are associated, and the two lines of text in area b-1 can be processed in an associated manner, while the two lines of text in area b-2 are not associated, and the two lines of text in area b-2 can be processed separately.
[0109] By using the image processing solution provided in the embodiment of the present disclosure to process the user medical record 700A, the following can be obtained: Figure 7c The translation result 700C shown in FIG. 7 is obtained by processing the user medical record 700A using the traditional solution. Figure 7d The translation result 700D is shown.
[0110] Comparing translation result 700C and translation result 700D, it can be seen that the typesetting method of translation result 700C is similar to user medical record 700A, and translation result 700D has non-standard problems in format, sentence, font, etc.
[0111] In summary, according to the embodiments of the present disclosure, multiple target areas in the image to be processed are identified, and the multiple target areas at least completely cover the corresponding text information in the image to be processed. The corresponding text information in the multiple target areas is identified through the trained recognition model. Based on the correlation between two adjacent text information in the corresponding text information, the corresponding text information is associated. The associated corresponding text information is translated from one language into another language to obtain the corresponding translated text information. In the image to be processed, the corresponding text information is replaced with the corresponding translated text information to obtain the target image. In this way, text information of any shape can be identified, and the adjacent text information identified is associated to achieve paragraph correction, thereby ensuring that the translated text information has clear paragraphs and smooth sentences. At the same time, the translated text information and the image are combined to optimize the typesetting method after translation. Further, in the medical field, the effect of translation can be significantly improved, which is conducive to doctors to better help patients view their conditions and reduce the risks of doctors and patients seeing doctors and taking medicines.
[0112] Figure 8 1 is a block diagram of an apparatus 800 for image processing according to some embodiments of the present disclosure. The apparatus 800 may be implemented in the server 120. Each module / component in the apparatus 800 may be implemented by hardware, software, firmware or any combination thereof.
[0113] The device 800 includes a first recognition module 810, which is configured to identify multiple target areas in the image to be processed in response to receiving the image to be processed, and the multiple target areas at least completely cover the corresponding text information in the image to be processed. The device 800 also includes a second recognition module 820, which is configured to identify the corresponding text information in the multiple target areas through a trained recognition model. The device 800 also includes an association module 830, which is configured to associate the corresponding text information based on the association degree of two adjacent text information in the corresponding text information. The device 800 also includes a translation module 840, which is configured to translate the associated corresponding text information from one language to another language to obtain the corresponding translated text information. The device 800 also includes a replacement module 850, which is configured to replace the corresponding text information with the corresponding translated text information in the image to be processed to obtain the target image.
[0114] In some embodiments, the first recognition module 810 is also configured to identify multiple candidate areas in the image to be processed, and the multiple candidate areas at least cover part of the corresponding text information; and adjust the identified multiple candidate areas so that the multiple candidate areas completely cover the corresponding text information to obtain multiple target areas.
[0115] In some embodiments, the association determination module of device 800 is configured to respectively determine the target participles in two adjacent text messages; and determine the ratio of the number of target participles appearing simultaneously in two adjacent text messages to the total number of target participles appearing in two adjacent text messages as the association.
[0116] In some embodiments, the replacement module 850 is further configured to remove the corresponding text information from the image to be processed, fill the multiple target areas with colors that coordinate with the surrounding images; and add the corresponding translated text information to the image to be processed.
[0117] In some embodiments, removing corresponding text information from the image to be processed and filling multiple target areas with colors coordinated with surrounding images are implemented based on a target strategy; the target strategy is determined as follows: based on the image to be processed, determining a first image and a second image, the first image being an image obtained by removing multiple target areas in the image to be processed, and the second image being an image obtained by filling multiple target areas with colors coordinated with surrounding images; determining a first feature representation of the first image, a second feature representation of the second image, and a third feature representation of multiple target areas in a predetermined feature space; determining a first distance between the first feature representation and the third feature representation; determining a second distance between the second feature representation and the third feature representation; and determining the target strategy based on the first distance, the second distance, and a predetermined threshold.
[0118] In some embodiments, the replacement module 850 is further configured to obtain a layout of the corresponding text information; adjust the corresponding translated text information based on the layout; and add the adjusted corresponding translated text information to the image to be processed.
[0119] In some embodiments, the replacement module 850 is further configured to determine position information of the plurality of target areas; and based on the position information, add corresponding translated text information to the image to be processed.
[0120] In some embodiments, the recognition model is trained based on data in a predetermined field.
[0121] In some embodiments, the target image needs to satisfy at least one of the following: the position of the corresponding translated text information in the target image is the same as the position of the corresponding text information in the image to be processed, the layout of the corresponding translated text information is the same as the layout of the corresponding text information, or the clarity of the target image satisfies a predetermined clarity.
[0122] The units included in the device 800 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 800 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0123] Fig. 9 900 is a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. It should be understood that Fig. 9 The electronic device 900 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Fig. 9 The electronic device 900 shown can be used to implement Figure 1 Server 120.
[0124] like Fig. 9As shown, the electronic device 900 is in the form of a general electronic device. The components of the electronic device 900 may include, but are not limited to, one or more processors or processing units 910, a memory 920, a storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. The processing unit 910 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 920. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 900.
[0125] The electronic device 900 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 930 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 900.
[0126] The electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig. 9 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 920 may include a computer program product 925 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.
[0127] The communication unit 940 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 900 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 900 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.
[0128] The input device 950 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 960 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 900 may also communicate with one or more external devices (not shown) through the communication unit 940 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 900, or communicate with any device that allows the electronic device 900 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).
[0129] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, wherein the one or more computer instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.
[0130] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0131] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0132] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0133] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.
[0134] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the implementations disclosed herein.
Claims
1. A method for image processing, comprising: In response to receiving the image to be processed, identifying a plurality of target areas in the image to be processed, wherein the plurality of target areas at least completely cover corresponding text information in the image to be processed; Recognize the corresponding text information in the plurality of target areas through the trained recognition model; Associating the corresponding text information based on the degree of association between two adjacent text information in the corresponding text information; Translating the associated corresponding text information from one language into another language to obtain corresponding translated text information; as well as In the image to be processed, the corresponding text information is replaced with the corresponding translated text information to obtain a target image.
2. The method according to claim 1, wherein identifying a plurality of target areas in the image to be processed comprises: Identifying a plurality of candidate regions in the image to be processed, wherein the plurality of candidate regions at least cover a portion of the corresponding text information; as well as The identified multiple candidate regions are adjusted so that the multiple candidate regions completely cover the corresponding text information, thereby obtaining the multiple target regions. 3 . The method according to claim 1 , wherein the two adjacent text information indicate two adjacent text information located in the same target area, or two adjacent text information located in different target areas.
4. The method according to claim 1, wherein associating the corresponding text information comprises: In response to the association degree exceeding an association degree threshold, the two adjacent text information are associated.
5. The method according to claim 1, further comprising: respectively determining target segmented words in the two adjacent text information; as well as The correlation degree is determined as the ratio of the number of simultaneous occurrences of the target participle in the two adjacent text information to the total number of occurrences of the target participle in the two adjacent text information.
6. The method according to claim 1, wherein replacing the corresponding text information with the corresponding translated text information comprises: removing the corresponding text information from the image to be processed, and filling the multiple target areas with colors that coordinate with surrounding images; as well as The corresponding translated text information is added to the image to be processed.
7. The method according to claim 6, wherein removing the corresponding text information from the image to be processed and filling the plurality of target areas with colors coordinated with surrounding images are achieved based on a target strategy; The target strategy is determined as follows: Based on the image to be processed, determining a first image and a second image, wherein the first image is an image obtained by removing the multiple target areas in the image to be processed, and the second image is an image obtained by filling the multiple target areas with colors coordinated with surrounding images; In a predetermined feature space, determining a first feature representation of the first image, a second feature representation of the second image, and a third feature representation of the plurality of target regions; determining a first distance between the first feature representation and the third feature representation; determining a second distance between the second feature representation and the third feature representation; and The target strategy is determined based on the first distance, the second distance, and a predetermined threshold.
8. The method according to claim 6, wherein adding the corresponding translated text information to the image to be processed comprises: Obtaining a typesetting method of the corresponding text information; Based on the typesetting method, adjusting the corresponding translated text information; as well as The adjusted corresponding translated text information is added to the image to be processed.
9. The method according to claim 6, wherein adding the corresponding translated text information to the image to be processed comprises: Determining location information of the multiple target areas; as well as Based on the position information, the corresponding translated text information is added to the image to be processed.
10. The method according to claim 1, wherein the recognition model is trained based on data in a predetermined field.
11. The method according to claim 1, wherein the target image needs to satisfy at least one of the following: The position of the corresponding translated text information in the target image is the same as the position of the corresponding text information in the image to be processed, The corresponding translated text information is laid out in the same manner as the corresponding text information, or The definition of the target image meets the predetermined definition.
12. An apparatus for image processing, comprising: A first recognition module is configured to, in response to receiving an image to be processed, identify a plurality of target areas in the image to be processed, wherein the plurality of target areas at least completely cover corresponding text information in the image to be processed; A second recognition module is configured to recognize the corresponding text information in the plurality of target areas through a trained recognition model; an associating module configured to associate the corresponding text information based on the degree of association between two adjacent text information in the corresponding text information; A translation module, configured to translate the associated corresponding text information from one language into another language to obtain corresponding translated text information; as well as The replacement module is configured to replace the corresponding text information with the corresponding translated text information in the image to be processed to obtain a target image.
13. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.
14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.
15. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 11.