Method and device for extracting reaction data from chemical reaction picture

Abstract

The invention provides a method and a device for extracting reaction data from a chemical reaction picture. The method comprises the following steps: acquiring the chemical reaction picture; identifying the chemical reaction picture to obtain arrow information and chemical element information in the chemical reaction picture; based on the chemical reaction pictures, the arrow information and the chemical element information, chemical reaction data corresponding to the chemical reaction pictures are extracted, and the chemical reaction data comprise reactants, products and reaction conditions of chemical reactions in the chemical reaction pictures.

Description

Technical Field

[0001] The present application relates to the field of data extraction technology, and in particular to a method and device for extracting reaction data from chemical reaction images. Background Art

[0002] In chemical literature and patent documents, chemical reactions are typically described in two ways: textual descriptions and chemical reaction scheme diagrams. Chemical reaction scheme diagrams can intuitively and accurately display complex chemical reaction pathways and the synthesis process of target products. Consequently, the academic community has gradually recognized the importance of converting these reaction diagrams into structured data. However, the diversity and complexity of chemical reaction scheme diagrams make it difficult to automatically identify and extract reaction data.

[0003] Therefore, how to accurately identify and extract reaction data in chemical reaction scheme diagrams has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of the embodiments of the present application is to provide a method and device for extracting reaction data from chemical reaction images, so as to solve the problems of time-consuming and labor-intensive and low success rate in generating chemical reactions in the prior art.

[0005] In a first aspect, embodiments of the present application provide a method for extracting reaction data from a chemical reaction image, comprising:

[0006] Get chemical reaction images;

[0007] Identify the chemical reaction image to obtain arrow information and chemical element information in the chemical reaction image;

[0008] Based on the chemical reaction image, the arrow information, and the chemical element information, chemical reaction data corresponding to the chemical reaction image is extracted, where the chemical reaction data includes reactants, products, and reaction conditions of the chemical reaction in the chemical reaction image.

[0009] In a possible implementation, identifying the chemical reaction image to obtain arrow information in the chemical reaction image includes:

[0010] The chemical reaction image is recognized by the first recognition model to obtain the arrow position and arrow shape of each arrow in the chemical reaction image.

[0011] In a possible implementation, identifying the chemical reaction image to obtain chemical element information in the chemical reaction image includes:

[0012] The chemical reaction image is identified by the second recognition model to obtain chemical elements in the chemical reaction image and the element position of each chemical element, wherein the chemical elements include molecules, labels, and texts.

[0013] In a possible implementation manner, after obtaining the chemical elements in the chemical reaction image, the method further includes:

[0014] Character recognition is performed on the label and the text respectively to obtain label content and text content.

[0015] In a possible implementation, extracting chemical reaction data corresponding to the chemical reaction image based on the chemical reaction image, the arrow information, and the chemical element information includes:

[0016] Processing the chemical reaction image, the arrow information, and the chemical element information through a third model to obtain a relative position relationship between the arrow and the chemical element;

[0017] Based on the relative position relationship, chemical reaction data corresponding to the chemical reaction image is extracted.

[0018] In a possible implementation, before processing the chemical reaction image, the arrow information, and the chemical element information using the third model, the method further includes:

[0019] Encoding the chemical reaction picture through an image encoder to obtain a vector corresponding to the chemical reaction picture;

[0020] The vector is mapped to the third model through a projection layer.

[0021] In a possible implementation, the step of training to obtain the third model includes:

[0022] Obtaining a training set, wherein the training set includes multiple groups of training data, each group of training data includes reaction image samples, arrow information samples, chemical element information samples, and theoretical reaction data;

[0023] The pre-built initial model is used to calculate the reaction image samples, arrow information samples, and chemical element information samples in each set of training data to obtain multiple sets of actual position relationships;

[0024] determining actual reaction data based on the actual position relationship;

[0025] The third model is obtained based on the actual reaction data and the theoretical reaction data.

[0026] In a second aspect, an embodiment of the present application further provides a device for extracting reaction data from a chemical reaction image, comprising:

[0027] an acquisition module configured to acquire a chemical reaction image;

[0028] an identification module configured to identify the chemical reaction image and obtain arrow information and chemical element information in the chemical reaction image;

[0029] A determination module is configured to extract chemical reaction data corresponding to the chemical reaction image based on the chemical reaction image, the arrow information, and the chemical element information, wherein the chemical reaction data includes reactants, products, and reaction conditions of the chemical reaction in the chemical reaction image.

[0030] In a third aspect, an embodiment of the present application further provides a storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for extracting reaction data from a chemical reaction image as described in any one of the above are executed.

[0031] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through a bus, and when the machine-readable instructions are executed by the processor, the steps of the method for extracting reaction data from a chemical reaction image as described in any one of the above are performed.

[0032] The embodiment of the present application first identifies the arrow information and chemical element information in the chemical reaction image, and then obtains the relative position relationship between the arrow and the chemical element based on the chemical reaction image, the arrow information and the chemical element information, and then extracts the chemical reaction data corresponding to the chemical reaction image based on the relative position relationship, and ensures the accuracy of the chemical reaction data.

[0033] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in this application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0035] Figure 1A flow chart showing a method for extracting reaction data from a chemical reaction image provided by the present application is shown;

[0036] Figure 2 A chemical reaction picture provided by this application is shown;

[0037] Figure 3 A flowchart of a training method for obtaining a third model provided by the present application is shown;

[0038] Figure 4 A schematic diagram of the structure of a device for extracting reaction data from a chemical reaction image provided by the present application is shown;

[0039] Figure 5 A schematic structural diagram of the electronic device provided in this application is shown. DETAILED DESCRIPTION

[0040] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0041] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0042] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0043] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0044] It should also be understood that although the present application has been described with reference to certain specific examples, those skilled in the art will be able to implement many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0045] The above and other aspects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0046] Specific embodiments of the present application will be described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments described are merely examples of the present application and may be implemented in a variety of ways. Familiar and / or repetitive functions and structures are not described in detail to avoid obscuring the present application with unnecessary or redundant details. Therefore, the specific structural and functional details described herein are not intended to be limiting, but rather serve merely as a basis and representative basis for the claims to teach those skilled in the art to variously utilize the present application with substantially any suitable detailed structure.

[0047] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," which may all refer to one or more of the same or different embodiments according to the present application.

[0048] First, in order to facilitate understanding of this application, a method for extracting reaction data from a chemical reaction image provided by this application is first described in detail. Figure 1 As shown, the method for extracting reaction data from a chemical reaction image provided by an embodiment of the present application includes steps S101-S103.

[0049] S101, obtaining a chemical reaction image.

[0050] As one example, Figure 2 A chemical reaction picture is shown, which may be captured from chemical literature, newspapers, etc., or may be captured by an image acquisition device, etc.

[0051] Specifically, the chemical reaction picture includes a complete chemical reaction, that is, the reactants involved in the chemical reaction, the reaction conditions required during the reaction, and the products generated after the reaction is completed.

[0052] S102: Recognize the chemical reaction image to obtain arrow information and chemical element information in the chemical reaction image.

[0053] After obtaining the chemical reaction image, the chemical reaction image is recognized to obtain arrow information and chemical element information in the chemical reaction image.

[0054] Specifically, the chemical reaction image can be identified once to obtain the arrow information and chemical element information, or the chemical reaction image can be identified twice to obtain the arrow information and chemical element information, that is, the chemical reaction image is identified once for the arrow and once for the chemical element. Compared with identifying the chemical reaction image twice to obtain the arrow information and chemical element information, identifying the chemical reaction image once to obtain the arrow information and chemical element information has a higher recognition efficiency; compared with identifying the chemical reaction image once to obtain the arrow information and chemical element information, identifying the chemical reaction image twice to obtain the arrow information and chemical element information has a higher recognition accuracy.

[0055] Next, the embodiment of the present application is described in detail by taking the example of setting up two recognitions of a chemical reaction image to obtain arrow information and chemical element information.

[0056] Optionally, when the chemical reaction picture is recognized to obtain arrow information in the chemical reaction picture, the chemical reaction picture is recognized using the first recognition model to obtain the arrow position and arrow shape of each arrow in the chemical reaction picture.

[0057] Here, the first recognition model is pre-trained and is used to recognize arrows in a chemical reaction image. As one example, the input of the first recognition model is a chemical reaction image, and the output of the first recognition model is the arrow position and arrow shape of each arrow in the chemical reaction image.

[0058] The arrow position represents the position of the arrow in the chemical reaction image and includes one or more coordinate values. The coordinate values ​​can be pixel coordinates or coordinates in a pre-established specific coordinate system. For example, when the coordinate values ​​are pixel coordinates, the pixel in the lower left corner of the chemical reaction image can be determined as the origin of the pixel coordinate system. Based on this, the coordinate value of each arrow is determined, specifically the coordinate value of the starting point of the arrow, the coordinate value of the ending point of the arrow, and the coordinate value of any point on the outline of the arrow. As long as the position of the arrow in the chemical reaction image can be represented, it will be sufficient.

[0059] Furthermore, the arrow form includes the shape, size, direction, type, etc. of the arrow, and the types of arrows include single-headed arrows, multi-headed arrows, unidirectional arrows, and bidirectional arrows, etc.

[0060] Of course, when training the first recognition model, the above information (chemical reaction images and arrow information) is also used as training data to train the initial model, so that a first recognition model with relatively accurate recognition can be obtained. The specific training steps are not described in detail here.

[0061] Furthermore, when the chemical reaction image is identified to obtain chemical element information in the chemical reaction image, the chemical reaction image is identified through the second recognition model to obtain the chemical elements in the chemical reaction image and the element position of each chemical element. The chemical elements include molecules, labels and text.

[0062] Similarly, the second recognition model is pre-trained and is used to identify chemical elements in a chemical reaction image. Accordingly, the input of the second recognition model is the chemical reaction image, and the output of the second recognition model is each chemical element and the element position of each chemical element in the chemical reaction image.

[0063] Reference Figure 2 In the chemical reaction picture shown, number A indicates a molecule, number B indicates a label, and number C indicates text. While identifying each chemical element, the element position of each chemical element is determined. Similarly, the element position of the chemical element characterizes the position of the chemical element in the chemical reaction picture. The element position of the chemical element includes one or more coordinate values, which can be pixel coordinates or coordinates in a pre-established specific coordinate system. When determining the element position of a chemical element, a corresponding rectangular frame is constructed for the chemical element according to the construction rules. The chemical element is located in the rectangular frame, and the coordinate values ​​of any one or more points of the rectangular frame corresponding to the chemical element are used as the coordinate values ​​included in the element position. Here, the construction rules include the distance between the chemical element and the like.

[0064] In another example, after obtaining the chemical elements in the chemical reaction image, character recognition is performed on the label and text respectively to obtain the label content and text content. Optionally, character recognition is performed on the label and text by an optical character recognition (OCR) model to obtain the label content and text content. The OCR model can adopt an open source solution or use specialized OCR data to train its own model according to actual conditions to accurately extract the label content and text content.

[0065] As another example, if a chemical reaction image is recognized once to obtain arrow information and chemical element information, the first recognition model and the second recognition model can be merged into a large recognition model. The input of the large recognition model is also the chemical reaction image, and the output of the large recognition model is the chemical elements such as arrows, molecules, labels, and text in the chemical reaction image. This application will not elaborate on this example.

[0066] It is worth noting that when obtaining the arrow information and chemical element information in the chemical reaction image, the chemical reaction image can be recognized by the first recognition model and the second recognition model, and the chemical reaction image can also be processed by image processing technology to obtain the arrow information and chemical element information. The embodiment of the present application does not make specific limitations on this.

[0067] S103, extracting chemical reaction data corresponding to the chemical reaction image based on the chemical reaction image, arrow information, and chemical element information. The chemical reaction data includes reactants, products, and reaction conditions of the chemical reaction in the chemical reaction image.

[0068] After obtaining the arrow information and chemical element information in the chemical reaction image, chemical reaction data corresponding to the chemical reaction image is further extracted based on the chemical reaction image, the arrow information and the chemical element information.

[0069] Optionally, when extracting chemical reaction data corresponding to the chemical reaction picture based on the chemical reaction picture, arrow information and chemical element information, the chemical reaction picture, arrow information and chemical element information are processed by a third model to obtain the relative position relationship between the arrow and the chemical element.

[0070] The arrow information and chemical element information in the embodiment of the present application are expressed in text. Based on this, the third model can be set to a large language model to ensure that the third model can more accurately identify the arrow information and chemical element information.

[0071] Furthermore, the third model also needs to process the chemical reaction image. Based on this, before the chemical reaction image, arrow information, and chemical element information are processed by the third model, the chemical reaction image is encoded by the image encoder to obtain the vector corresponding to the chemical reaction image, and the vector is mapped to the third model through the projection layer so that the third model can accurately obtain the information of the chemical reaction image. Among them, the image encoder of the embodiment of the present application can be set to a commonly used visual encoder such as ViT and CLIP.

[0072] After the relative position relationship between the arrow and the chemical element is obtained through the third model, the chemical reaction data corresponding to the chemical reaction image is extracted based on the relative position relationship. The relative position relationship between the arrow and the chemical element includes the chemical element being located on the side of the starting point of the arrow, the chemical element being located on the side of the ending point of the arrow, and the chemical element being located on either side except the beginning and end of the arrow. Figure 2In the chemical reaction image shown, the relative positional relationship between molecule A and the target arrow is that the chemical element is located on one side of the arrow's starting point, the relative positional relationship between molecule E and the target arrow is that the chemical element is located on one side of the arrow's ending point, and the relative positional relationship between text C and the target arrow is that the chemical element is located on either side of the arrow except the starting and ending sides. It should be noted that the target arrow in the figure is the arrow indicated by number D.

[0073] The chemical reaction data extracted based on the relative position relationship between the arrows and the chemical elements can accurately describe the chemical reaction expressed by the chemical reaction picture, ensuring the accuracy of the chemical reaction data and saving time for subsequent chemical reaction research.

[0074] In the method of the embodiment of the present application, the arrow position and arrow shape of each arrow in the chemical reaction image are identified by the first recognition model, and the chemical elements in the chemical reaction image and the element position of each chemical element are identified by the second recognition model, thereby achieving the purpose of accurately identifying, segmenting and extracting arrows and chemical elements in the reaction image; further, the chemical reaction image, arrow information and chemical element information are processed by the third model to determine the relative position relationship between the arrow and the chemical element, so that the third model can accurately locate and understand the reaction process, and then output the chemical reaction data corresponding to the chemical reaction image, which greatly enhances the accuracy, practicality and comprehensiveness of the analysis of the chemical reaction image.

[0075] Optionally, Figure 3 A flow chart of a method for training to obtain a third model is shown, and specific steps include S301-S304.

[0076] S301, obtaining a training set, where the training set includes multiple groups of training data, each group of training data includes reaction image samples, arrow information samples, chemical element information samples, and theoretical reaction data.

[0077] S302: Calculate the reaction image samples, arrow information samples, and chemical element information samples in each set of training data using a pre-built initial model to obtain multiple sets of actual position relationships.

[0078] S303: Determine actual response data based on the actual position relationship.

[0079] S304: Obtain a third model based on the actual reaction data and the theoretical reaction data.

[0080] The training data in the training set is manually screened and annotated. For example, for a reaction image sample, arrows, chemical elements, and other information are pre-annotated. At the same time, arrow information samples corresponding to each arrow and chemical element information samples corresponding to each chemical element are set. Furthermore, the reaction image sample is described to obtain its corresponding theoretical reaction data.

[0081] Next, the reaction image samples, arrow information samples, chemical element information samples, and theoretical reaction data from each training data set are sequentially fed into a pre-built initial model. The initial model calculates the reaction image samples, arrow information samples, and chemical element information samples to determine the actual positional relationships corresponding to the reaction image samples in that training data set—that is, the relative positional relationships between the arrows and chemical elements in the reaction image samples.

[0082] Furthermore, the actual reaction data is determined based on the actual position relationship, and then the similarity value between the actual reaction data and the theoretical reaction data is calculated. If the similarity value is less than the preset threshold, the model parameters are adjusted and the above training steps are performed again until the similarity value is greater than or equal to the preset threshold, thereby obtaining the third model.

[0083] Here, when the third model is set as a large language model, you can choose an open source multimodal large language model solution with similar functions, such as Mini GPTv2, Cog VLM-Grounding, Ferretv2, Llama3, and Mistral. Of course, you can also choose a model with different parameter amounts according to the hardware situation. This embodiment of the application is not specifically limited to this.

[0084] Based on the same inventive concept, the second aspect of this application also provides a device for extracting reaction data from chemical reaction images. Since the principle of solving the problem by the device in this application is similar to the above-mentioned method of extracting reaction data from chemical reaction images in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0085] See also Figure 4 As shown, the device for extracting reaction data from a chemical reaction image includes:

[0086] An acquisition module 401 is configured to acquire a chemical reaction image;

[0087] an identification module 402 configured to identify the chemical reaction image and obtain arrow information and chemical element information in the chemical reaction image;

[0088] The determination module 403 is configured to extract chemical reaction data corresponding to the chemical reaction image based on the chemical reaction image, the arrow information, and the chemical element information. The chemical reaction data includes reactants, products, and reaction conditions of the chemical reaction in the chemical reaction image.

[0089] In another embodiment, when the recognition module 402 recognizes the chemical reaction image and obtains the arrow information in the chemical reaction image, the body configuration is as follows:

[0090] The chemical reaction image is recognized by the first recognition model to obtain the arrow position and arrow shape of each arrow in the chemical reaction image.

[0091] In another embodiment, when the recognition module 402 recognizes the chemical reaction image and obtains the chemical element information in the chemical reaction image, it is configured to:

[0092] The chemical reaction image is identified by the second recognition model to obtain chemical elements in the chemical reaction image and the element position of each chemical element, wherein the chemical elements include molecules, labels, and texts.

[0093] In yet another embodiment, the identification module 402 is further configured to:

[0094] Character recognition is performed on the label and the text respectively to obtain label content and text content.

[0095] In yet another embodiment, the determining module 403 is specifically configured as follows:

[0096] Processing the chemical reaction image, the arrow information, and the chemical element information through a third model to obtain a relative position relationship between the arrow and the chemical element;

[0097] Based on the relative position relationship, chemical reaction data corresponding to the chemical reaction image is extracted.

[0098] In yet another embodiment, the apparatus for extracting reaction data from a chemical reaction image further includes an encoding module 404 configured to:

[0099] Encoding the chemical reaction picture through an image encoder to obtain a vector corresponding to the chemical reaction picture;

[0100] The vector is mapped to the third model through a projection layer.

[0101] In yet another embodiment, the apparatus for extracting reaction data from a chemical reaction image further includes a training module 405 configured to:

[0102] Obtaining a training set, wherein the training set includes multiple groups of training data, each group of training data includes reaction image samples, arrow information samples, chemical element information, and theoretical reaction data;

[0103] The pre-built initial model is used to calculate the reaction image samples, arrow information samples, and chemical element information in each set of training data to obtain multiple sets of actual position relationships;

[0104] determining actual reaction data based on the actual position relationship;

[0105] The third model is obtained based on the actual reaction data and the theoretical reaction data.

[0106] The embodiment of the present application first identifies the arrow information and chemical element information in the chemical reaction image, and then obtains the relative position relationship between the arrow and the chemical element based on the chemical reaction image, the arrow information and the chemical element information, and then extracts the chemical reaction data corresponding to the chemical reaction image based on the relative position relationship, and ensures the accuracy of the chemical reaction data.

[0107] A third aspect of the present application further provides a storage medium, which is a computer-readable medium and stores a computer program. When the computer program is executed by a processor, the method provided in any embodiment of the present application is implemented, including the following steps:

[0108] S11, obtain chemical reaction images;

[0109] S12, identifying the chemical reaction image to obtain arrow information and chemical element information in the chemical reaction image;

[0110] S13, extracting chemical reaction data corresponding to the chemical reaction image based on the chemical reaction image, the arrow information, and the chemical element information, where the chemical reaction data includes reactants, products, and reaction conditions of the chemical reaction in the chemical reaction image.

[0111] The embodiment of the present application first identifies the arrow information and chemical element information in the chemical reaction image, and then obtains the relative position relationship between the arrow and the chemical element based on the chemical reaction image, the arrow information and the chemical element information, and then extracts the chemical reaction data corresponding to the chemical reaction image based on the relative position relationship, and ensures the accuracy of the chemical reaction data.

[0112] It should be noted that the storage medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any storage medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the storage medium may be conveyed using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), or any suitable combination thereof.

[0113] The fourth aspect of the present application also provides an electronic device, such as Figure 5 As shown, the electronic device includes at least a memory 501 and a processor 502. The memory 501 stores a computer program. The processor 502 implements the method provided by any embodiment of the present application when executing the computer program on the memory 501. Exemplarily, the method executed by the electronic device computer program is as follows:

[0114] S21, obtaining chemical reaction images;

[0115] S22, identifying the chemical reaction image to obtain arrow information and chemical element information in the chemical reaction image;

[0116] S23, extracting chemical reaction data corresponding to the chemical reaction image based on the chemical reaction image, the arrow information, and the chemical element information, where the chemical reaction data includes reactants, products, and reaction conditions of the chemical reaction in the chemical reaction image.

[0117] The embodiment of the present application first identifies the arrow information and chemical element information in the chemical reaction image, and then obtains the relative position relationship between the arrow and the chemical element based on the chemical reaction image, the arrow information and the chemical element information, and then extracts the chemical reaction data corresponding to the chemical reaction image based on the relative position relationship, and ensures the accuracy of the chemical reaction data.

[0118] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0119] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

[0120] In addition, although adopting specific order to describe each operation, this should not be interpreted as requiring these operations to be executed in the specific order shown or in sequential order.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be interpreted as limiting the scope of the application.Some features described in the context of separate embodiment can also be implemented in a single embodiment in combination.On the contrary, the various features described in the context of a single embodiment also can be implemented in multiple embodiments individually or in the mode of any suitable subcombination.

[0121] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0122] The above describes in detail several embodiments of the present application, but the present application is not limited to these specific embodiments. Based on the concept of the present application, technicians in this field can make various variations and modifications to the embodiments, and these variations and modifications should fall within the scope of protection required by the present application.

Claims

1. A method for extracting reaction data from a chemical reaction image, characterized in that: include: Get chemical reaction images; Identify the chemical reaction image to obtain arrow information and chemical element information in the chemical reaction image; Based on the chemical reaction image, the arrow information, and the chemical element information, chemical reaction data corresponding to the chemical reaction image is extracted, where the chemical reaction data includes reactants, products, and reaction conditions of the chemical reaction in the chemical reaction image.

2. The method according to claim 1, characterized in that Identifying the chemical reaction image to obtain arrow information in the chemical reaction image includes: The chemical reaction image is recognized by the first recognition model to obtain the arrow position and arrow shape of each arrow in the chemical reaction image.

3. The method according to claim 1, characterized in that Identifying the chemical reaction image to obtain chemical element information in the chemical reaction image includes: The chemical reaction image is identified by the second recognition model to obtain chemical elements in the chemical reaction image and the element position of each chemical element, wherein the chemical elements include molecules, labels, and texts.

4. The method according to claim 3, characterized in that After obtaining the chemical elements in the chemical reaction picture, the method further includes: Character recognition is performed on the label and the text respectively to obtain label content and text content.

5. The method according to claim 1, wherein The extracting chemical reaction data corresponding to the chemical reaction picture based on the chemical reaction picture, the arrow information, and the chemical element information includes: Processing the chemical reaction image, the arrow information, and the chemical element information through a third model to obtain a relative position relationship between the arrow and the chemical element; Based on the relative position relationship, chemical reaction data corresponding to the chemical reaction image is extracted.

6. The method according to claim 5, characterized in that Before processing the chemical reaction picture, the arrow information, and the chemical element information by the third model, the method further includes: Encoding the chemical reaction picture through an image encoder to obtain a vector corresponding to the chemical reaction picture; The vector is mapped to the third model through a projection layer.

7. The method according to claim 5, characterized in that The step of training to obtain the third model includes: Obtaining a training set, wherein the training set includes multiple groups of training data, each group of training data includes reaction image samples, arrow information samples, chemical element information samples, and theoretical reaction data; The pre-built initial model is used to calculate the reaction image samples, arrow information samples, and chemical element information samples in each set of training data to obtain multiple sets of actual position relationships; determining actual reaction data based on the actual position relationship; The third model is obtained based on the actual reaction data and the theoretical reaction data.

8. A device for extracting reaction data from a chemical reaction image, characterized in that: include: an acquisition module configured to acquire a chemical reaction image; an identification module configured to identify the chemical reaction image and obtain arrow information and chemical element information in the chemical reaction image; A determination module is configured to extract chemical reaction data corresponding to the chemical reaction image based on the chemical reaction image, the arrow information, and the chemical element information, wherein the chemical reaction data includes reactants, products, and reaction conditions of the chemical reaction in the chemical reaction image.

9. A storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for extracting reaction data from a chemical reaction image according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via a bus. When the machine-readable instructions are executed by the processor, the steps of the method for extracting reaction data from a chemical reaction image as described in any one of claims 1 to 7 are performed.

Images