A petri dish colony map generation method and apparatus

Abstract

The application discloses a culture dish colony map generation method and device, and relates to the technical field of image generation. The culture dish colony map generation method comprises the following steps: obtaining a target colony description text and a target culture dish background image; generating a target low-resolution colony map according to the target colony description text; generating a target high-resolution colony map according to the target low-resolution colony map; and fusing the target high-resolution colony map and the target culture dish background image to obtain a target high-resolution culture dish colony map. The culture dish colony map disclosed by the application can automatically generate a high-resolution culture dish colony map required by a user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation, in particular to a culture dish colony map generation method and device. BACKGROUND

[0002] In some application scenarios, a user needs to use a culture dish colony map. At present, the culture dish colony map is obtained by manually collecting the culture dish colony map through a collection device. However, in some cases, due to some factors (such as scratches, light source pollution, artifacts, and dense colonies), it is difficult to collect the culture dish colony map required by the user. SUMMARY

[0003] Therefore, the present application provides a culture dish colony map generation method and device to generate the culture dish colony map required by the user, and the technical solutions are as follows.

[0004] The first aspect of the present application provides a culture dish colony map generation method, comprising:

[0005] obtaining a target colony description text and a target culture dish background image;

[0006] generating a target low-resolution colony map according to the target colony description text;

[0007] generating a target high-resolution colony map according to the target low-resolution colony map;

[0008] fusing the target high-resolution colony map and the target culture dish background image to obtain a target high-resolution culture dish colony map.

[0009] In a possible implementation, the generating a target low-resolution colony map according to the target colony description text comprises:

[0010] using a pre-constructed low-resolution colony map generation model to generate a target low-resolution colony map according to the target colony description text;

[0011] wherein the low-resolution colony map generation model uses a generation network in a first generative adversarial network, the first generative adversarial network is trained using a first training data set, and the first training data set includes training colony description texts and real low-resolution colony maps corresponding to the training colony description texts;

[0012] The training target of the generation network in the first generative adversarial network includes: making the low-resolution colony map generated by the generation network in the first generative adversarial network according to the training colony description text consistent with the real low-resolution colony map corresponding to the training colony description text.

[0013] In a possible implementation, the low-resolution colony map generation model is used to generate a target low-resolution colony map based on the target colony description text, and the method comprises the following steps of:

[0014] encoding the target colony description text to obtain a feature vector of the target colony description text;

[0015] extracting style features from the feature vector of the target colony description text by using the pre-constructed low-resolution colony map generation model, wherein the low-resolution colony map generation model extracts style features from the feature vector of the target colony description text based on a multi-head attention mechanism to obtain style features of the target colony description text;

[0016] generating a target low-resolution colony map based on the style features of the target colony description text by using the low-resolution colony map generation model.

[0017] In a possible implementation, the low-resolution colony map generation model comprises a style feature acquisition module for extracting style features from the feature vector of the target colony description text.

[0018] The style feature acquisition module comprises a normalization layer, M cascaded processing modules, and M multi-head attention modules, wherein each processing module comprises one or more cascaded first fully connected layers, the feature vector of the target colony description text is input into the normalization layer, the output of the normalization layer is input into a first processing module in the M cascaded processing modules, the M cascaded processing modules correspond to the M multi-head attention modules one by one, the output of a processing module is input into a multi-head attention module, the style feature acquisition module further comprises a second fully connected layer, the outputs of the M multi-head attention modules are input into the second fully connected layer, and the output of the second fully connected layer is the style features of the target colony description text, M is an integer greater than 1.

[0019] In a possible implementation, the target high-resolution colony map is generated based on the target low-resolution colony map, and the method comprises the following steps of:

[0020] generating a target high-resolution colony map based on the target low-resolution colony map by using a pre-constructed high-resolution colony map generation model;

[0021] wherein the high-resolution colony map generation model adopts a generation network in a second generative adversarial network trained by using second training data in a second training data set, the second training data comprises training low-resolution colony maps and real high-resolution colony maps corresponding to the training low-resolution colony maps;

[0022] The training target of the generation network in the second generative adversarial network comprises: making the high-resolution colony map generated by the generation network in the second generative adversarial network according to the training low-resolution colony map consistent with the real high-resolution colony map corresponding to the training low-resolution colony map.

[0023] In a possible implementation, the training target of the second generative adversarial network further comprises:

[0024] making the high-level features of the high-resolution colony map generated by the generation network in the second generative adversarial network according to the training low-resolution colony map approach the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map, wherein the high-level features are features capable of representing structural information of the colony in the image;

[0025] and / or making the texture features of the high-resolution colony map generated by the generation network in the second generative adversarial network according to the training low-resolution colony map approach the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map.

[0026] In a possible implementation, the fusing of the target high-resolution colony map and the target petri dish background map to obtain the target high-resolution petri dish colony map comprises:

[0027] fusing the target high-resolution colony map and the target petri dish background map to obtain the target high-resolution petri dish colony map by using a pre-constructed image fusion model;

[0028] wherein the image fusion model adopts a generation network in a third generative adversarial network trained by third training data in a third training data set, the third training data comprises a training image pair composed of a training petri dish background map and a training high-resolution colony map and a real high-resolution petri dish colony map corresponding to the training image pair, and the third training data set contains training petri dish background maps of multiple styles;

[0029] The training target of the generation network in the third generative adversarial network comprises: making the high-resolution petri dish colony map generated by the generation network in the third generative adversarial network according to the training image pair consistent with the real high-resolution petri dish colony map corresponding to the training image pair.

[0030] In a possible implementation, the training target of the third generative adversarial network further comprises:

[0031] making the style features of the high-resolution petri dish colony map generated by the generation network in the third generative adversarial network according to the training image pair approach the style features of the real high-resolution petri dish colony map corresponding to the training image pair.

[0032] In a possible implementation, the image fusion model comprises a style feature acquisition module configured to acquire the style feature of the target high-resolution colony map and the style feature of the target Petri dish background map, and an image generation module configured to generate the target high-resolution Petri dish colony map according to the style feature of the target high-resolution colony map and the style feature of the target Petri dish background map.

[0033] The image generation module comprises a plurality of cascaded image generation sub-modules. The input of the first image generation sub-module is the fused feature obtained by fusing the style feature of the target high-resolution colony map and the style feature of the target Petri dish background map. The input of any non-first image generation sub-module is the fused feature and the output of the previous image generation sub-module. The output of the last image generation sub-module is the target high-resolution Petri dish colony map. Each image generation sub-module comprises an enhanced convolution module.

[0034] The second aspect of the application provides a Petri dish colony map generation device, comprising a data acquisition module, a low-resolution colony map generation module, a high-resolution colony map generation module, and an image fusion module.

[0035] The data acquisition module is configured to acquire a target colony description text and a target Petri dish background map.

[0036] The low-resolution colony map generation module is configured to generate a target low-resolution colony map according to the target colony description text.

[0037] The high-resolution colony map generation module is configured to generate a target high-resolution colony map according to the target low-resolution colony map.

[0038] The image fusion module is configured to fuse the target high-resolution colony map and the target Petri dish background map to obtain a target high-resolution Petri dish colony map.

[0039] According to the above technical solution, the Petri dish colony map generation method provided by the application first acquires a target colony description text and a target Petri dish background map, then generates a target low-resolution colony map according to the target colony description text, then generates a target high-resolution colony map according to the target low-resolution colony map, and finally generates a target high-resolution Petri dish colony map according to the target Petri dish background map and the target high-resolution colony map. The Petri dish colony map generation method provided by the application can automatically generate a high-resolution Petri dish colony map that meets the user's needs. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only only the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0041] Figure 1 A schematic diagram of a system architecture related to the present application;

[0042] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a flow of a culture dish colony map generation method provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a flow of generating a high-resolution culture dish colony map based on multiple models provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of a structure of a low-resolution colony map generation model provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a structure of an image fusion model provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of a structure of a culture dish colony map generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. The terms used in the implementation part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0050] The embodiments of the present application will be described below in conjunction with the accompanying drawings. Those skilled in the art can know that with the development of technology and the appearance of new scenes, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0051] The terms "first", "second", and the like in the description and in the claims of the present application and above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the terms so used are interchangeable under appropriate circumstances and are merely employed to distinguish one object from another. Furthermore, the terms "comprise", "have", and any variations thereof are intended to cover a non-exclusive inclusion, such that processes, methods, systems, products, or devices that comprise a list of elements are not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such processes, methods, systems, products, or devices.

[0052] Before introducing the petri dish colony map generation method provided by the present application, the system architecture involved in the present application is described.

[0053] In one possible implementation, as shown in Figure 1 The system architecture involved in the present application can include a terminal 101 and a server 102, and the terminal 101 can interact with the server 102 through a network (wired network or wireless network). The server 102 can include one or more servers (one server is taken as an example for description in the following description). Figure 1 The terminal 101 can obtain a target colony description text and a target petri dish background image, and transmit the target colony description text and the target petri dish background image to the server 102 through the network. The server 102 generates a target high-resolution petri dish colony map by using the petri dish colony map generation method provided by the present application, and transmits the target high-resolution petri dish colony map to the terminal 101 through the network. The terminal 101 can output the target high-resolution petri dish colony map.

[0054] In another possible implementation, the system architecture involved in the present application can include a terminal. The terminal has strong data processing capability. The terminal can generate a target high-resolution petri dish colony map by using the petri dish colony map generation method provided by the present application, and then output the target high-resolution petri dish colony map.

[0055] Next, the product form of the terminal is described.

[0056] The terminal described above can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like, and the embodiments of the present application do not make any limitation in this regard.

[0057] Figure 2 An optional hardware structure diagram of the terminal is shown.

[0058] Reference Figure 2 As shown, the terminal can include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), a headphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290, and the like. Those skilled in the art can understand that the terminal is not limited to the components shown in the figure, and can include more or fewer components, or combine certain components, or different components. Figure 2 The above is only an example of the terminal and does not constitute a limitation on the terminal, which can include more or fewer components than shown, or combine certain components, or different components.

[0059] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the terminal. Specifically, the input unit 230 can include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect user touch operations (such as user operations on or near the touch screen using fingers, joints, stylus, etc.) on or near it, and drive the corresponding connection device according to the pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive commands from the processor 270 and execute them; the touch signal at least includes touch point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, the other input devices 232 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, on-off buttons, etc.), trackballs, mice, joysticks, etc.

[0060] The display unit 240 can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file displays, and / or playing of any kind of multimedia files.

[0061] The memory 220 can be used to store instructions and data. The memory 220 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc. The storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc. or their subsets, extended sets. Non-volatile random access memory can also be included. The processor 270 is provided with software and applications that include management of hardware, software and data resources in the computing processing device, support control. It is also used for the storage of multimedia files, as well as the storage of running programs and applications.

[0062] The processor 270 is the control center of the terminal, which connects various parts of the terminal through various interfaces and lines, executes various functions of the terminal and processes data by running or executing instructions stored in the memory 220 and calling data stored in the memory 220, thereby overall controlling the terminal. Optionally, the processor 270 can include one or more processing units. Preferably, the processor 270 can integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 270. In some embodiments, the processor, the memory, can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips respectively. The processor 270 can also be used to generate corresponding operation control signals to the corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0063] The memory 220 can be used to store instructions and data. The memory 220 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc. The storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc. or their subsets, extended sets. Non-volatile random access memory can also be included. The processor 270 is provided with software and applications that include management of hardware, software and data resources in the computing processing device, support control. It is also used for the storage of multimedia files, as well as the storage of running programs and applications.

[0064] The radio frequency unit 210 (optional) can be used for receiving and sending signals in the process of information or communication, for example, receiving the downlink information of the base station and processing by the processor 270; in addition, sending the uplink data to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0065] In the embodiments of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, which can be replaced by other communication interfaces, for example, can be a network interface.

[0066] The terminal also includes a power supply 290 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management, etc. through the power management system.

[0067] The terminal also includes an external interface 280, which can be a standard Micro USB interface, or can be a multi-pin connector, and can be used for connecting the terminal with other devices for communication, or can be used for connecting a charger to charge the terminal.

[0068] Although not shown, the terminal can also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, different function sensors, etc., which will not be described here.

[0069] Next, the product form of the above server is described.

[0070] Figure 3 A structural schematic diagram of the above server is provided, as shown in Figure 3As shown, the server can include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate through the bus 301.

[0071] The bus 301 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0072] The processor 302 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0073] The memory 304 can include a volatile memory, such as a random access memory (RAM). The memory 304 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard disk drive (HDD), or a solid state drive (SSD).

[0074] The memory 304 can be used to store software code related to the petri dish colony map generation method, and the processor 302 can call the software code stored in the memory 304, or can schedule other units to realize the corresponding functions.

[0075] The processor in the terminal and the server (for example, the processor 270 and the processor 302) can be a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and the like), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, and the like, or a hardware system without an instruction execution function, such as an ASIC, an FPGA, and the like, or a combination of the hardware system without the instruction execution function and the hardware system with the instruction execution function.

[0076] Next, the Petri dish colony map generation method provided in the present application is introduced through the following examples.

[0077] Referring to Figure 4 , a flowchart of the Petri dish colony map generation method provided in the embodiments of the present application is shown, which can include the following steps:

[0078] Step S401: Obtain a target colony description text and a target Petri dish background image.

[0079] The target colony description text is a description text of a colony in a Petri dish colony map required by a user, and the target colony description text describes colony related information, wherein the colony related information can include part or all of the following information: species, quantity, morphology, size, distribution, unit density, whether there is an artifact, whether there is light source pollution, and the like.

[0080] For example, the target colony description text is "probiotics, about 100 in quantity, two small pixels, five large pixels, and random distribution".

[0081] The target Petri dish background image is an image of a Petri dish in a Petri dish colony map required by a user, which is a background image of the colony described in the target colony description text.

[0082] Step S402: Generate a target low-resolution colony map according to the target colony description text.

[0083] After obtaining the target colony description text, the present embodiment first generates a low-resolution colony map consistent with the target colony description text according to the target colony description text, that is, a target low-resolution colony map.

[0084] In one possible implementation manner, as Figure 5As shown, the pre-constructed low-resolution colony map generation model can be used to generate a target low-resolution colony map based on the target colony description text.

[0085] The low-resolution colony map generation model is trained by using training colony description texts and real low-resolution colony maps corresponding to the training colony description texts.

[0086] Step S403: generating a target high-resolution colony map based on the target low-resolution colony map.

[0087] After obtaining the target low-resolution colony map that matches the content of the target colony description text, the target low-resolution colony map is further processed into a target high-resolution colony map that matches the content of the target colony description text.

[0088] In one possible implementation, as shown in Figure 5 As shown, the pre-constructed high-resolution colony map generation model can be used to generate a target high-resolution colony map based on the target low-resolution colony map.

[0089] The high-resolution colony map generation model is trained by using training low-resolution colony maps and real high-resolution colony maps corresponding to the training low-resolution colony maps.

[0090] Step S404: fusing the target high-resolution colony map and a target petri dish background map to obtain a target high-resolution petri dish colony map.

[0091] After obtaining the target high-resolution colony map, the target high-resolution colony map can be fused with the target petri dish background map to obtain a target high-resolution petri dish colony map.

[0092] In one possible implementation, as shown in Figure 5 As shown, the pre-constructed image fusion model can be used to fuse the target high-resolution colony map and the target petri dish background map to obtain a target high-resolution petri dish colony map.

[0093] The image fusion model is trained by using training image pairs composed of training petri dish background maps and training high-resolution colony maps, and real high-resolution petri dish colony maps corresponding to the training image pairs.

[0094] The culture dish colony map generation method provided in the embodiments of the present application first acquires a target colony description text and a target culture dish background image, then generates a target low-resolution colony map according to the target colony description text, then generates a target high-resolution colony map according to the target low-resolution colony map, and finally generates a target high-resolution culture dish colony map according to the target culture dish background image and the target high-resolution colony map. The culture dish colony map generation method provided in the embodiments of the present application can automatically generate a high-resolution culture dish colony map meeting the user demand.

[0095] In another embodiment of the present application, the specific implementation process of "step S402: generating a target low-resolution colony map according to a target colony description text" in the above embodiment is introduced.

[0096] The above embodiment mentions that the pre-constructed low-resolution colony map generation model can be used to generate a target low-resolution colony map according to a target colony description text.

[0097] In a possible implementation manner, the low-resolution colony map generation model adopts a generation network in a first generative adversarial network (for example, StyleGAN) trained.

[0098] The generation network in the first generative adversarial network is trained by using first training data in a first training data set. The first training data set contains a plurality of first training data, and each first training data includes a training colony description text and a real low-resolution colony map corresponding to the training colony description text.

[0099] The training target of the generation network in the first generative adversarial network can include: making the low-resolution colony map generated by the generation network in the first generative adversarial network according to the training colony description text consistent with the real low-resolution colony map corresponding to the training colony description text.

[0100] In a possible implementation manner, the process of generating a target low-resolution colony map according to a target colony description text by using a pre-constructed low-resolution colony map generation model can include:

[0101] Step a1, encoding the target colony description text to obtain a feature vector of the target colony description text.

[0102] In a possible implementation manner, the BERT model (a language representation model) can be used to encode the target colony description text to obtain the feature vector of the target colony description text. It should be noted that the present embodiment does not limit the use of the BERT model to obtain the feature vector of the target colony description text, and any language model that can obtain the feature vector of the target colony description text is applicable to the present application.

[0103] Step a2, the style feature of the feature vector of the target colony description text is obtained by using the pre-constructed low-resolution colony map generation model, and the style feature of the target colony description text is obtained.

[0104] The style feature is a feature capable of representing the style.

[0105] The feature vector of the target colony description text can be input into the pre-constructed low-resolution colony map generation model. The low-resolution colony map generation model first obtains the style feature of the feature vector of the target colony description text based on the multi-head attention mechanism. The multi-head attention mechanism can capture the details of the target colony description text description in different representation subspaces.

[0106] Step a3, the target low-resolution colony map is generated based on the style feature of the target colony description text by using the low-resolution colony map generation model.

[0107] The low-resolution colony map generation model in this embodiment can include a style feature acquisition module for acquiring the style feature from the feature vector of the target colony description text, and an image generation module for generating a low-resolution colony map based on the style feature acquired by the style feature acquisition module.

[0108] The style feature acquisition module of the low-resolution colony map generation model can include a normalization layer, M cascaded processing modules, and M multi-head attention modules. Each processing module includes one or more cascaded first fully connected layers. The feature vector of the target colony description text is input into the normalization layer, and the output of the normalization layer is input into the first processing module of the M cascaded processing modules. The M cascaded processing modules correspond to the M multi-head attention modules one by one. The output of a processing module is input into a multi-head attention module. The style feature acquisition module further includes a second fully connected layer. The outputs of the M multi-head attention modules are input into the second fully connected layer, and the second fully connected layer outputs the style feature of the target colony description text. M is an integer greater than 1.

[0109] The image generation module of the low-resolution colony map generation model includes a plurality of cascaded image generation sub-modules. The input of the first image generation sub-module is the style feature of the target colony description text. The input of each subsequent image generation sub-module is the style feature of the target colony description text and the output of the previous image generation sub-module. The output of the last image generation sub-module is the final low-resolution colony map.

[0110] Please refer to Figure 6 , which shows an example of a low-resolution colony map generation model, as Figure 6As shown, the style feature acquisition module of the low-resolution colony image generation model includes a normalization layer, eight cascaded first fully connected layers (FC), a first multi-head attention module, a second multi-head attention module, and a second fully connected layer. The feature vector of the target colony description text (e.g., a 512×1 vector) is input to the normalization layer for normalization processing. The normalization result passes through the eight first fully connected layers in sequence. The outputs of the first four first fully connected layers are input to the first multi-head attention module for attention calculation, and the first multi-head attention module outputs the first attention feature. The outputs of the last four first fully connected layers are input to the second multi-head attention module for attention calculation, and the second multi-head attention module outputs the second attention feature. The first and second attention features are input to the second fully connected layer for processing, and the second fully connected layer outputs the style feature of the target colony description text (e.g., a 512×1 vector).

[0111] When any multi-head attention module performs attention calculation on the input, it first generates a query vector, key vector, and value vector for each attention head based on the input. Each attention head performs attention calculation on its own query vector, key vector, and value vector. Assuming there are h attention heads, h attention features can be obtained through the above process. After obtaining h attention features, the h attention features can be concatenated to obtain the final attention features.

[0112] The i-th attention head is based on the following equations (1) and (2) for the query vector Q. i Key vector K i Sum vector V i Attention is calculated to obtain the attention feature. i

[0113] (1)

[0114] (2)

[0115] Among them, score(Q) i ,K ij α represents the attention score. ij Let V represent the attention weights, and n represent the value vector V. i The dimension of the key vector K i The dimension is also n.

[0116] like Figure 6As shown, the image generation module of the low-resolution colony map generation model includes a plurality of cascaded image generation sub-modules, the first image generation sub-module includes two instance normalization modules (AdaIN, Adaptive Instance Normalization) and a convolution module (such as a 3x3 convolution module), the style feature input of the target colony description text is processed by the first instance normalization module of the first image generation sub-module, the processing result of the first instance normalization module is input into the convolution module for processing, the result of the convolution module processing and the style feature of the target colony description text are input into the second instance normalization module for processing, and the processing result of the second instance normalization module is the final output of the first image generation sub-module. Each image generation sub-module includes an upsampling module (for improving resolution), two instance normalization modules and a convolution module. For any non-first image generation sub-module, the output of the previous image generation sub-module is input into the upsampling module for upsampling processing, the upsampling result is input into the first instance normalization module for processing, the processing result of the first instance normalization module is input into the convolution module for processing, the processing result of the convolution module and the style feature of the target colony description text are input into the second instance normalization module for processing, and the processing result of the second instance normalization module is the final output of the non-first image generation sub-module.

[0117] The above mentioned that the first generative adversarial network is trained by using the training colony description text and the real low-resolution colony map corresponding to the training colony description text. When training the generation network in the first generative adversarial network, first, the feature vector of the training colony description text can be obtained, and then the feature vector of the training colony description text is input into the generation network in the first generative adversarial network. The generation network generates a low-resolution colony map, and then the low-resolution colony map generated by the generation network and the real low-resolution colony map corresponding to the training colony description text are input into the discriminant network in the first generative adversarial network to obtain the discrimination result of the low-resolution colony map generated by the generation network and the discrimination result of the real low-resolution colony map corresponding to the training colony description text. Then, the first adversarial loss is determined according to the discrimination result of the generated low-resolution colony map and the discrimination result of the real low-resolution colony map corresponding to the training colony description text. Finally, the first generative adversarial network is updated according to the first adversarial loss. Wherein, the first adversarial loss L GAN1 The calculation method of (G, D) is as follows:

[0118] (3)

[0119] Wherein, x1 represents the real low-resolution colony map corresponding to the training colony description text, p data(x1) represents a distribution of x1, D(x1) represents a discrimination result of the discrimination network in the first generative adversarial network on x1, G(z1) represents a low-resolution colony map generated by the generation network in the first generative adversarial network according to the training colony description text, and D(G(z1)) represents a discrimination result of the discrimination network in the first generative adversarial network on G(z1).

[0120] In another possible implementation, in addition to determining the first adversarial loss, a matching loss of the low-resolution colony map generated by the generation network and the training colony description text can be determined, the matching loss is fused with the first adversarial loss, and then the first generative adversarial network is updated in parameters according to the fused loss.

[0121] In another embodiment of the present application, the specific implementation process of "step S403: generating a target high-resolution colony map according to a target low-resolution colony map" in the above embodiment is introduced.

[0122] The above embodiment mentions that the target high-resolution colony map can be generated based on a pre-constructed high-resolution colony map generation model, and the target low-resolution colony map is taken as a generation basis.

[0123] In a possible implementation, the high-resolution colony map generation model can adopt a generation network in a second generative adversarial network (for example, a super-resolution generative adversarial network SRGANs) trained.

[0124] In a possible implementation, the training target of the second generative adversarial network can include: making the high-resolution colony map generated by the generation network in the second generative adversarial network according to the training low-resolution colony map consistent with the real high-resolution colony map corresponding to the training low-resolution colony map.

[0125] To improve the quality of the high-resolution colony map generated by the second generative adversarial network, in another possible implementation, the training target of the generation network in the second generative adversarial network can include: (1) causing the generation network in the second generative adversarial network to make the high-resolution colony map generated by the generation network from the training low-resolution colony map consistent with the real high-resolution colony map corresponding to the training low-resolution colony map; (2) causing the generation network in the second generative adversarial network to make the high-level features of the high-resolution colony map generated by the generation network from the training low-resolution colony map consistent with the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map; (3) causing the generation network in the second generative adversarial network to make the texture features of the high-resolution colony map generated by the generation network from the training low-resolution colony map consistent with the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map. It should be noted that the training target of the generation network in the second generative adversarial network can also include any one of (2) and (3) above, and (1) above.

[0126] Specifically, when training the generation network in the second generative adversarial network: input the training low-resolution colony map into the generation network in the second generative adversarial network, and the generation network generates a high-resolution colony map according to the input; input the high-resolution colony map generated by the generation network and the real high-resolution colony map corresponding to the training low-resolution colony map into the discriminant network in the second generative adversarial network, to obtain the discriminant result of the high-resolution colony map generated by the generation network and the discriminant result of the real high-resolution colony map corresponding to the training low-resolution colony map, and determine the second adversarial loss according to the discriminant result of the high-resolution colony map generated by the generation network and the discriminant result of the real high-resolution colony map corresponding to the training low-resolution colony map; obtain the high-level features of the generated high-resolution colony map and the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map based on the pre-trained network, and determine the perception loss according to the high-level features of the generated high-resolution colony map and the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map; obtain the texture features of the generated high-resolution colony map and the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map; determine the texture loss according to the texture features of the generated high-resolution colony map and the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map; fuse the second adversarial loss, the perception loss and the texture loss, and update the parameters of the second generative adversarial network according to the fused loss. The high-level features are features that can represent the colony structure information in the image.

[0127] The second adversarial loss L GAN2 The calculation method of (G, D) is as follows:

[0128] (4)

[0129] wherein x2 represents a real high-resolution colony map corresponding to the training low-resolution colony map, D(x2) represents a discrimination result of the discrimination network in the second generative adversarial network on x2, G(z2) represents a high-resolution colony map generated by the generation network in the second generative adversarial network according to the training low-resolution colony map, and D(G(z2)) represents a discrimination result of the discrimination network in the second generative adversarial network on G(z2).

[0130] perception loss L perceptual The calculation manner of (G) is as follows:

[0131] (5)

[0132] wherein, represents a high-level feature of G(z2), represents a high-level feature of x2.

[0133] texture loss L texture The calculation manner of (G) is as follows:

[0134] (6)

[0135] wherein, represents a texture feature extraction function, represents a high-level feature extracted texture feature, i.e., a texture feature of G(z2), represents a high-level feature extracted texture feature, i.e., a texture feature of x2. It should be noted that the texture loss represents a difference between the texture feature of the generated high-resolution colony map and the texture feature of the real high-resolution colony map.

[0136] After obtaining the second adversarial loss L GAN2 (G,D), the perception loss L perceptual (G), and the texture loss L texture (G), the three losses can be fused in a manner shown in the following formula to obtain a final loss L final (G,D):

[0137] (7)

[0138] wherein, and are weight coefficients, which can be set according to specific conditions, for example, the following can be set: and are both 0.5.

[0139] In another embodiment of the present application, the implementation process of "step S404: fusing the target high-resolution colony map and the target petri dish background map to obtain a target high-resolution petri dish colony map" in the above embodiment is introduced.

[0140] The above embodiment mentions that the target high-resolution colony map and the target petri dish background map can be fused by using a pre-constructed image fusion model to obtain a target high-resolution petri dish colony map.

[0141] In a possible implementation, the image fusion model can use a generative network in a third generative adversarial network trained. The third generative adversarial network is trained by using third training data in a third training data set, and the third training data includes training image pairs composed of training petri dish background maps and training high-resolution colony maps, and real high-resolution petri dish colony maps corresponding to the training image pairs. The third training data set can include training petri dish background maps of various styles, such as images of petri dishes with different culture liquids, images of petri dishes with scratches, images of petri dishes with covers, images of petri dishes with labels, images of petri dishes with pen marks, images of petri dishes with condensed water vapor, and the like.

[0142] The training target of the generative network in the third generative adversarial network includes: making the high-resolution petri dish colony map generated by the generative network in the third generative adversarial network according to the training image pair consistent with the real high-resolution petri dish colony map corresponding to the training image pair.

[0143] In a possible implementation, the process of fusing the target high-resolution colony map and the target petri dish background map by using the pre-constructed image fusion model to obtain a target high-resolution petri dish colony map can include:

[0144] Step b1, respectively encoding the target petri dish background map and the target high-resolution colony map to obtain a feature vector of the target petri dish background map and a feature vector of the target high-resolution colony map.

[0145] The existing image encoder can be used to respectively encode the target petri dish background map and the target high-resolution colony map to obtain the feature vector of the target petri dish background map and the feature vector of the target high-resolution colony map.

[0146] Step b2, using the pre-constructed image fusion model to extract style features from the feature vector of the target petri dish background map and the feature vector of the target high-resolution colony map to obtain a style feature of the target petri dish background map and a style feature of the target high-resolution colony map.

[0147] Step b3, using the image fusion model, taking the style features of the target Petri dish background image and the style features of the target high-resolution colony image as the basis, to generate a target high-resolution Petri dish colony image.

[0148] The image fusion model in this embodiment can include a style feature acquisition module and an image generation module. Please refer to Figure 7 , which shows an example of an image fusion model, as Figure 7 shown, the style feature acquisition module in the image fusion model can include a normalization layer and a plurality of cascaded fully connected layers (such as 8 cascaded fully connected layers). The feature vector of the target Petri dish background image (such as a 512x1 vector) is input into the normalization layer for normalization processing, and the normalized processing result is sequentially processed by the 8 cascaded fully connected layers. The last fully connected layer outputs the style features of the target Petri dish background image (such as a 512x1 vector). Similarly, the feature vector of the target high-resolution colony image (such as a 512x1 vector) is input into the normalization layer for normalization processing, and the normalized processing result is sequentially processed by the plurality of cascaded fully connected layers. The last fully connected layer outputs the style features of the target high-resolution colony image (such as a 512x1 vector).

[0149] After obtaining the style features of the target Petri dish background image and the style features of the target high-resolution colony image, the two style features are fused, and the fused features are input into the image generation module of the image fusion model, as Figure 7As shown, the image generation module of the image fusion model can include a plurality of cascaded image generation sub-modules, the input of the first image generation sub-module is the fused feature, the input of each image generation sub-module is the fused feature and the output of the previous image generation sub-module, and the output of the last image generation sub-module is the target high-resolution petri dish colony map. Among them, the first image generation sub-module includes two instance normalization modules and an enhanced convolution module, the first instance normalization module of the first image generation sub-module processes the input fused feature, the processing result of the first instance normalization module is input into the enhanced convolution module for processing, the processing result of the enhanced convolution module and the fused feature are input into the second instance normalization module for processing, and the processing result of the second instance normalization module is the final output of the first image generation sub-module. Each image generation sub-module includes an upsampling module (for improving resolution), two instance normalization modules and an enhanced convolution module. For any non-first image generation sub-module, the output of the previous image generation sub-module is input into the upsampling module for upsampling processing, the upsampling result is input into the first instance normalization module for processing, the processing result of the first instance normalization module is input into the enhanced convolution module for processing, the processing result of the enhanced convolution module and the fused feature are input into the second instance normalization module for processing, and the processing result of the second instance normalization module is the final output of the non-first image generation sub-module.

[0150] It should be noted that the enhanced convolution module is used to improve the resolution and detail capture capability of the generated image, which can be achieved by adding more convolution layers and residual connections in the convolution operation:

[0151] (8)

[0152] Among them, Conv(x) represents a convolution operation, and x represents an input image.

[0153] It is mentioned above that the third generative adversarial network is trained by using the training image pair composed of the training culture dish background image and the training high-resolution colony image and the real high-resolution culture dish colony image corresponding to the training image pair. When training the generative network in the third generative adversarial network, the training image pair can be input into the generative network in the third generative adversarial network. The generative network generates a high-resolution culture dish colony image according to the input. The generated high-resolution culture dish colony image and the real high-resolution culture dish colony image corresponding to the training image pair are input into the discriminative network in the third generative adversarial network. The discriminative results of the high-resolution culture dish colony image generated by the generative network and the real high-resolution culture dish colony image corresponding to the training image pair are obtained. The third adversarial loss is determined according to the discriminative results of the high-resolution culture dish colony image generated by the generative network and the real high-resolution culture dish colony image corresponding to the training image pair. The third generative adversarial network is updated according to the third adversarial loss. The calculation method of the third adversarial loss is similar to the calculation methods of the first adversarial loss and the second adversarial loss, which will not be described herein.

[0154] In order to obtain a model with better performance, in another possible implementation, in addition to determining the third adversarial loss, a style mixing loss can also be determined for the generated high-resolution culture dish colony image and the real high-resolution culture dish colony image corresponding to the training image pair. Then, the style mixing loss and the third adversarial loss are fused, and the third generative adversarial network is updated according to the fused loss. The style features of the high-resolution culture dish colony image generated by the generative network in the third generative adversarial network according to the training image pair tend to be close to the style features of the real high-resolution culture dish colony image corresponding to the training image pair.

[0155] The determination method of the style mixing loss Style Loss is as follows:

[0156] (9)

[0157] wherein, the mean of the style features of the high-resolution culture dish colony image G(z3) generated by the generative network in the third generative adversarial network, the mean of the style features of the real high-resolution culture dish colony image corresponding to the training image pair, the variance of the style features of the high-resolution culture dish colony image G(z3) generated by the generative network in the third generative adversarial network, the variance of the style features of the real high-resolution culture dish colony image corresponding to the training image pair.

[0158] The culture dish colony map generation method provided by the embodiments of the present application can generate a high-resolution culture dish colony map that meets user requirements, is relatively real, and is fine in details based on three models.

[0159] The culture dish colony map generation method provided by the embodiments of the present application is introduced above, and the device corresponding to the culture dish colony map generation method is introduced below.

[0160] Please refer to Figure 8 , Figure 8 The culture dish colony map generation device provided by the embodiments of the present application has a structure diagram, and the culture dish colony map generation device can include a data acquisition module 801, a low-resolution colony map generation module 802, a high-resolution colony map generation module 803, and an image fusion module 804.

[0161] The data acquisition module 801 is configured to acquire target colony description text and a target culture dish background image.

[0162] The low-resolution colony map generation module 802 is configured to generate a target low-resolution colony map according to the target colony description text.

[0163] The high-resolution colony map generation module 803 is configured to generate a target high-resolution colony map according to the target low-resolution colony map.

[0164] The image fusion module 804 is configured to fuse the target high-resolution colony map and the target culture dish background image to obtain a target high-resolution culture dish colony map.

[0165] In a possible implementation, when the low-resolution colony map generation module 802 generates the target low-resolution colony map according to the target colony description text, the low-resolution colony map generation module 802 is specifically configured to:

[0166] use a pre-constructed low-resolution colony map generation model to generate the target low-resolution colony map according to the target colony description text;

[0167] The low-resolution colony map generation model uses a generation network in a first generative adversarial network that is trained using a first training data set, and the first training data set includes training colony description text and a real low-resolution colony map corresponding to the training colony description text.

[0168] The training target of the generation network in the first generative adversarial network includes: making the low-resolution colony map generated by the generation network in the first generative adversarial network according to the training colony description text consistent with the real low-resolution colony map corresponding to the training colony description text.

[0169] In a possible implementation, the low-resolution colony map generation module 802, when generating the target low-resolution colony map by using the pre-constructed low-resolution colony map generation model and taking the target colony description text as the generation basis, is specifically configured to:

[0170] encode the target colony description text to obtain a feature vector of the target colony description text;

[0171] extract style features from the feature vector of the target colony description text by using the pre-constructed low-resolution colony map generation model, wherein the low-resolution colony map generation model extracts the style features from the feature vector of the target colony description text based on a multi-head attention mechanism to obtain style features of the target colony description text;

[0172] generate the target low-resolution colony map by using the low-resolution colony map generation model and taking the style features of the target colony description text as the basis.

[0173] In a possible implementation, the low-resolution colony map generation model includes a style feature acquisition module configured to extract the style features from the feature vector of the target colony description text.

[0174] The style feature acquisition module includes a normalization layer, M cascaded processing modules, and M multi-head attention modules, wherein each processing module includes one or more cascaded first fully connected layers, the feature vector of the target colony description text is input into the normalization layer, the output of the normalization layer is input into a first processing module in the M cascaded processing modules, the M cascaded processing modules correspond to the M multi-head attention modules one by one, the output of a processing module is input into a multi-head attention module, the style feature acquisition module further includes a second fully connected layer, the outputs of the M multi-head attention modules are input into the second fully connected layer, and the output of the second fully connected layer is the style features of the target colony description text, M is an integer greater than 1.

[0175] In a possible implementation, the high-resolution colony map generation module 803, when generating the target high-resolution colony map according to the target low-resolution colony map, is specifically configured to:

[0176] generate the target high-resolution colony map by using the pre-constructed high-resolution colony map generation model and taking the target low-resolution colony map as the generation basis.

[0177] The high-resolution colony map generation model adopts a generation network in a second generative adversarial network trained by a second training data set, the second generative adversarial network is trained by second training data in the second training data set, and the second training data includes training low-resolution colony maps and real high-resolution colony maps corresponding to the training low-resolution colony maps.

[0178] The training target of the generative network in the second generative adversarial network comprises: making the high-resolution colony map generated by the generative network in the second generative adversarial network according to the training low-resolution colony map consistent with the real high-resolution colony map corresponding to the training low-resolution colony map.

[0179] In a possible implementation, the training target of the second generative adversarial network further comprises:

[0180] making the high-level features of the high-resolution colony map generated by the generative network in the second generative adversarial network according to the training low-resolution colony map approach the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map, wherein the high-level features are features capable of representing structural information of the colony in the image;

[0181] and / or making the texture features of the high-resolution colony map generated by the generative network in the second generative adversarial network according to the training low-resolution colony map approach the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map.

[0182] In a possible implementation, when the image fusion module 804 fuses the target high-resolution colony map and the target petri dish background map to obtain the target high-resolution petri dish colony map, the image fusion module 804 is specifically configured to:

[0183] fuse the target high-resolution colony map and the target petri dish background map to obtain the target high-resolution petri dish colony map by using a pre-constructed image fusion model;

[0184] wherein the image fusion model adopts the generative network in the third generative adversarial network, the third generative adversarial network is trained by using third training data in a third training data set, the third training data comprises a training image pair composed of a training petri dish background map and a training high-resolution colony map and a real high-resolution petri dish colony map corresponding to the training image pair, and the third training data set contains training petri dish background maps of multiple styles.

[0185] The training target of the generative network in the third generative adversarial network comprises: making the high-resolution petri dish colony map generated by the generative network in the third generative adversarial network according to the training image pair consistent with the real high-resolution petri dish colony map corresponding to the training image pair.

[0186] In a possible implementation, the training target of the third generative adversarial network comprises: making the style features of the high-resolution petri dish colony map generated by the generative network in the third generative adversarial network according to the training image pair approach the style features of the real high-resolution petri dish colony map corresponding to the training image pair.

[0187] In a possible implementation, the image fusion model comprises a style feature acquisition module configured to acquire the style feature of the target high-resolution colony map and the style feature of the target Petri dish background map, and an image generation module configured to generate the target high-resolution Petri dish colony map according to the style feature of the target high-resolution colony map and the style feature of the target Petri dish background map.

[0188] The image generation module comprises a plurality of cascaded image generation sub-modules. The input of a first image generation sub-module is a fused feature obtained by fusing the style feature of the target high-resolution colony map and the style feature of the target Petri dish background map. The input of any non-first image generation sub-module is the fused feature and the output of the previous image generation sub-module. The output of a last image generation sub-module is the target high-resolution Petri dish colony map. Each image generation sub-module comprises an enhanced convolution module.

[0189] The Petri dish colony map generation apparatus provided in the embodiments of the present application can automatically generate a high-resolution Petri dish colony map that meets user requirements and is relatively real and detailed.

[0190] The embodiments of the present application further provide an electronic device, which can comprise at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0191] In the embodiments of the present application, the number of processors, communication interfaces, memories and communication buses is at least one, and the processors, communication interfaces and memories communicate with each other through the communication bus.

[0192] The processor can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0193] The memory can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0194] The memory stores a program, and the processor can invoke the program stored in the memory. The program is configured to implement the steps of the Petri dish colony map generation method provided in the above embodiments.

[0195] The embodiments of the present application further provide a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the Petri dish colony map generation method provided in the above embodiments.

[0196] The embodiment of the present application further provides a computer program product comprising computer readable instructions which, when run on an electronic device, cause the electronic device to implement the steps of the culture dish colony map generation method provided by the above embodiment.

[0197] In addition, it should be noted that the above-described apparatus embodiments are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiments provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, training device, or network device, etc.) execute the methods described in various embodiments of the present application.

[0199] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of a computer program product in whole or in part.

[0200] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A method of generating a petri dish colony map, the method comprising: include: Obtain the target colony description text and the target culture dish background image. The target culture dish background image includes at least one of the following: images of culture dishes with different culture media, images of culture dishes with scratches, images of culture dishes with lids, images of culture dishes with labels, images of culture dishes with pen marks, images of culture dishes with artifacts, images of culture dishes with light source contamination, and images of culture dishes with condensation. Generate a target low-resolution colony map based on the target colony description text; Generate a target high-resolution colony map based on the target low-resolution colony map; The process of fusing the target high-resolution colony image with the target petri dish background image to obtain a target high-resolution petri dish colony image includes: using a pre-constructed image fusion model to fuse the target high-resolution colony image with the target petri dish background image to obtain a target high-resolution petri dish colony image; wherein, the image fusion model adopts the generative network in a trained third generative adversarial network, the third generative adversarial network is trained using a third training dataset, and the third training dataset contains training petri dish background images of various styles, specifically including at least one of the following: images of petri dishes with different culture media, images of petri dishes with scratches, images of petri dishes with lids, images of petri dishes with labels, images of petri dishes with pen marks, images of petri dishes with artifacts, images of petri dishes with light source contamination, and images of petri dishes with condensation; The step of generating a target high-resolution colony map based on the target low-resolution colony map includes: Using a pre-constructed high-resolution colony map generation model, a target high-resolution colony map is generated based on the target low-resolution colony map; wherein, the high-resolution colony map generation model adopts the generator network in the trained second generative adversarial network; When training the generative network in the second generative adversarial network: The low-resolution colony map is trained and input into the generator network in the second generative adversarial network, which generates a high-resolution colony map based on the input. The high-resolution colony map generated by the generator network and the real high-resolution colony map corresponding to the low-resolution colony map trained are input into the discriminator network in the second generative adversarial network to obtain the discrimination results of the high-resolution colony map generated by the generator network and the discrimination results of the real high-resolution colony map corresponding to the low-resolution colony map trained. The second adversarial loss is determined based on the discrimination results of the high-resolution colony map generated by the generator network and the discrimination results of the real high-resolution colony map corresponding to the low-resolution colony map trained. The high-level features of the generated high-resolution colony map and the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map are obtained based on the high-level features of the generated high-resolution colony map and the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map are determined. The high-level features are those that can characterize the colony structure information in the image. The texture features of the generated high-resolution colony map and the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map are obtained. The texture loss is determined based on the texture features of the generated high-resolution colony map and the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map. The second adversarial loss, the perceptual loss, and the texture loss are fused, and the parameters of the second generative adversarial network are updated based on the fused loss.

2. The petri dish colony map generation method according to claim 1, characterized by, The step of generating a target low-resolution colony map based on the target colony description text includes: Using a pre-built low-resolution colony map generation model, and based on the target colony description text, a target low-resolution colony map is generated. The low-resolution colony image generation model uses the generator network in the first generator adversarial network that has been trained. The first generator adversarial network is trained using the first training data in the first training dataset. The first training data includes training colony description text and the real low-resolution colony image corresponding to the training colony description text. The training objective of the generator network in the first generative adversarial network includes: making the low-resolution colony map generated by the generator network in the first generative adversarial network based on the training colony description text consistent with the real low-resolution colony map corresponding to the training colony description text.

3. The petri dish colony map generation method according to claim 2, characterized by, The step of generating a target low-resolution colony map using a pre-built low-resolution colony map generation model, based on the target colony description text, includes: The target colony description text is encoded to obtain the feature vector of the target colony description text; Using a pre-built low-resolution colony map generation model, style features are extracted from the feature vector of the target colony description text. The low-resolution colony map generation model extracts style features from the feature vector of the target colony description text based on a multi-head attention mechanism, thereby obtaining the style features of the target colony description text. Using the low-resolution colony map generation model, a target low-resolution colony map is generated based on the style features of the target colony description text.

4. The petri dish colony map generation method according to claim 3, characterized by, The low-resolution colony map generation model includes a style feature acquisition module for extracting style features from the feature vector of the target colony description text; The style feature acquisition module includes a normalization layer, M cascaded processing modules, and M multi-head attention modules. Each processing module includes one or more cascaded first fully connected layers. The feature vector of the target colony description text is input to the normalization layer, and the output of the normalization layer is input to the first processing module among the M cascaded processing modules. The M cascaded processing modules correspond one-to-one with the M multi-head attention modules, and the output of one processing module is input to one multi-head attention module. The style feature acquisition module also includes a second fully connected layer. The outputs of the M multi-head attention modules are input to the second fully connected layer, and the output of the second fully connected layer is the style feature of the target colony description text. M is an integer greater than 1.

5. The petri dish colony map generation method according to claim 1, characterized by, The second generative adversarial network is trained using the second training data in the second training dataset, which includes training low-resolution colony images and the real high-resolution colony images corresponding to the training low-resolution colony images. The training objective of the generator network in the second generative adversarial network includes: making the high-resolution colony map generated by the generator network in the second generative adversarial network based on the training low-resolution colony map as consistent with the real high-resolution colony map corresponding to the training low-resolution colony map.

6. The petri dish colony map generation method according to claim 5, characterized by, The training objectives of the second generative adversarial network also include: The high-level features of the high-resolution colony map generated by the generative network in the second generative adversarial network based on the training low-resolution colony map are made to approach the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map, wherein the high-level features are features that can characterize the structural information of colonies in the image. And / or, to make the texture features of the high-resolution colony map generated by the generative network in the second generative adversarial network based on the training low-resolution colony map approximate the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map.

7. The petri dish colony map generation method according to claim 1, characterized by, The third generative adversarial network is trained using a third training dataset, including: the third generative adversarial network is trained using third training data in the third training dataset, the third training data including training image pairs consisting of training petri dish background images and training high-resolution colony images, and real high-resolution petri dish colony images corresponding to the training image pairs. The training objective of the generator network in the third generative adversarial network includes: making the high-resolution culture dish colony image generated by the generator network in the third generative adversarial network based on the training image consistent with the real high-resolution culture dish colony image corresponding to the training image pair.

8. The petri dish colony map generation method according to claim 7, characterized by, The training objectives of the third generative adversarial network also include: The generative network in the third generative adversarial network is made to make the style features of the generated high-resolution petri dish colony image based on the training image approximate the style features of the corresponding real high-resolution petri dish colony image based on the training image.

9. The method for generating colony maps in petri dishes according to claim 7, characterized in that, The image fusion model includes a style feature acquisition module for acquiring style features of the target high-resolution colony image and style features of the target petri dish background image, and an image generation module for generating the target high-resolution petri dish colony image based on the style features of the target high-resolution colony image and the style features of the target petri dish background image. The image generation module includes multiple cascaded image generation sub-modules. The input of the first image generation sub-module is the fused feature obtained by fusing the style features of the target high-resolution colony image with the style features of the target petri dish background image. The input of any image generation sub-module other than the first one is the fused feature and the output of the previous image generation sub-module. The output of the last image generation sub-module is the target high-resolution petri dish colony image. Each image generation sub-module includes an enhanced convolution module.

10. A device for generating colony maps on petri dishes, characterized in that, include: The system includes a data acquisition module, a low-resolution colony image generation module, a high-resolution colony image generation module, and an image fusion module. The data acquisition module is used to acquire the target colony description text and the target culture dish background image. The target culture dish background image includes at least one of the following: images of culture dishes with different culture media, images of culture dishes with scratches, images of culture dishes with lids, images of culture dishes with labels, images of culture dishes with pen marks, images of culture dishes with artifacts, images of culture dishes with light source contamination, and images of culture dishes with condensation. The low-resolution colony image generation module is used to generate a target low-resolution colony image based on the target colony description text. The high-resolution colony map generation module is used to generate a target high-resolution colony map based on the target low-resolution colony map. The image fusion module is used to fuse the target high-resolution colony image with the target petri dish background image to obtain the target high-resolution petri dish colony image; The image fusion module is specifically used to: fuse the target high-resolution colony image with the target petri dish background image using a pre-built image fusion model to obtain the target high-resolution petri dish colony image; wherein, the image fusion model adopts the generative network in the trained third generative adversarial network, the third generative adversarial network is trained using a third training dataset, and the third training dataset contains training petri dish background images of various styles, specifically including at least one of the following: images of petri dishes with different culture media, images of petri dishes with scratches, images of petri dishes with lids, images of petri dishes with labels, images of petri dishes with pen marks, images of petri dishes with artifacts, images of petri dishes with light source contamination, and images of petri dishes with condensation; The high-resolution colony map generation module is specifically used for: Using a pre-constructed high-resolution colony map generation model, a target high-resolution colony map is generated based on the target low-resolution colony map; wherein, the high-resolution colony map generation model adopts the generator network in the trained second generative adversarial network; When training the generative network in the second generative adversarial network: The low-resolution colony map is trained and input into the generator network in the second generative adversarial network, which generates a high-resolution colony map based on the input. The high-resolution colony map generated by the generator network and the real high-resolution colony map corresponding to the low-resolution colony map trained are input into the discriminator network in the second generative adversarial network to obtain the discrimination results of the high-resolution colony map generated by the generator network and the discrimination results of the real high-resolution colony map corresponding to the low-resolution colony map trained. The second adversarial loss is determined based on the discrimination results of the high-resolution colony map generated by the generator network and the discrimination results of the real high-resolution colony map corresponding to the low-resolution colony map trained. The high-level features of the generated high-resolution colony map and the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map are obtained based on the high-level features of the generated high-resolution colony map and the high-level features of the real high-resolution colony map corresponding to the training low-resolution colony map are determined. The high-level features are those that can characterize the colony structure information in the image. The texture features of the generated high-resolution colony map and the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map are obtained. The texture loss is determined based on the texture features of the generated high-resolution colony map and the texture features of the real high-resolution colony map corresponding to the training low-resolution colony map. The second adversarial loss, the perceptual loss, and the texture loss are fused, and the parameters of the second generative adversarial network are updated based on the fused loss.

Images