Image reconstruction method, device, equipment, medium and product for vision abnormalities

Abstract

This disclosure provides embodiments of image reconstruction methods, apparatuses, devices, media, and products for visual impairments. One specific implementation of the method includes: determining a description of the visual impairment and a lesion image corresponding to a user with visual impairment, wherein the lesion image corresponds to the description of the visual impairment; determining a pre-trained lesion image generation model corresponding to the description of the visual impairment, wherein the lesion image generation model includes a generator, and the generator corresponds to a pre-generated inverse generator; generating a reconstructed image based on the lesion image and the inverse generator; and displaying the reconstructed image. This implementation can overcome the limitations of traditional visual aid device upgrades and iterations, improving the experience for users with visual impairments.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to image reconstruction methods, apparatus, devices, media, and products for visual impairments. Background Technology

[0002] Low vision typically refers to corrected visual acuity below 0.1. Various eye diseases, such as cataracts, glaucoma, and macular degeneration, are key causes of low vision. Currently, visual aids for low vision are still at the level of simple mechanical devices, and there are technical limitations in upgrading and iterating these devices.

[0003] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure provide image reconstruction methods, apparatuses, electronic devices, computer-readable media, and computer program products for visual impairments to address one or more of the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide an image reconstruction method for visual impairment, the method comprising: determining a visual impairment description and a lesion image corresponding to a user with visual impairment, wherein the lesion image corresponds to the visual impairment description; determining a pre-trained lesion image generation model corresponding to the visual impairment description, wherein the lesion image generation model includes a generator, and the generator corresponds to a pre-generated inverse generator; generating a reconstructed image based on the lesion image and the inverse generator; and displaying the reconstructed image.

[0007] Optionally, displaying the reconstructed image includes displaying the reconstructed image on a head-mounted display device corresponding to the visually impaired user.

[0008] Optionally, determining the vision abnormality description and lesion image for the corresponding user with vision abnormality includes: determining a normal image to be displayed; performing text encoding processing on the vision abnormality description to obtain text features; and inputting the normal image and the text features into the generator to obtain the lesion image.

[0009] Optionally, the above-mentioned text encoding process for the description of vision abnormality to obtain text features includes: converting the description of vision abnormality into a target language to obtain converted text; and inputting the converted text into a text encoder to obtain text features.

[0010] Optionally, the above-mentioned lesion image generation model further includes a discriminator, which includes a global image encoder and a local image encoder, and the global image encoder and the local image encoder are independent of each other in the network structure of the discriminator.

[0011] Optionally, the discriminator further includes a multi-head discriminator and a downsampling module. The multi-head discriminator takes the output of the local image encoder as input, and the downsampling module takes the output of the global image encoder as input. The network parameters of the global image encoder and the local image encoder are frozen.

[0012] Optionally, the generator includes a convolutional layer, a downsampling module, and a deep semantic editing network, wherein the deep semantic editing network includes an upsampling module, and the upsampling module includes an upsampling layer, a convolutional layer, and an affine layer.

[0013] Optionally, the aforementioned inverse generator is generated through the following steps: obtaining a set of lesion images corresponding to the aforementioned visual abnormality description; constructing an initial autoencoder network based on the aforementioned generator, wherein the encoder of the aforementioned initial autoencoder network is the aforementioned generator, the network parameters of the aforementioned encoder are fixed, and the decoder of the aforementioned initial autoencoder network is the inverse mapping of the aforementioned generator; training the aforementioned initial autoencoder network based on the aforementioned lesion image set to obtain an autoencoder network; and determining the decoder included in the aforementioned autoencoder network as the inverse generator.

[0014] Secondly, some embodiments of this disclosure provide an image reconstruction apparatus for visual impairment, the apparatus comprising: a first determining unit configured to determine a description of visual impairment and a lesion image corresponding to a user with visual impairment, wherein the lesion image corresponds to the description of visual impairment; a second determining unit configured to determine a pre-trained lesion image generation model corresponding to the description of visual impairment, wherein the lesion image generation model includes a generator, and the generator corresponds to a pre-generated inverse generator; a generation unit configured to generate a reconstructed image based on the lesion image and the inverse generator; and a display unit configured to display the reconstructed image.

[0015] Optionally, the display unit is further configured to display the reconstructed image in a head-mounted display device corresponding to the aforementioned visually impaired user.

[0016] Optionally, the first determining unit is further configured to: determine a normal image to be displayed; perform text encoding processing on the above-mentioned visual abnormality description to obtain text features; and input the above-mentioned normal image and the above-mentioned text features into the above-mentioned generator to obtain a lesion image.

[0017] Optionally, the first determining unit is further configured to: convert the above-mentioned vision abnormality description into the target language to obtain converted text; and input the above-mentioned converted text into a text encoder to obtain text features.

[0018] Optionally, the above-mentioned lesion image generation model further includes a discriminator, which includes a global image encoder and a local image encoder, and the global image encoder and the local image encoder are independent of each other in the network structure of the discriminator.

[0019] Optionally, the discriminator further includes a multi-head discriminator and a downsampling module. The multi-head discriminator takes the output of the local image encoder as input, and the downsampling module takes the output of the global image encoder as input. The network parameters of the global image encoder and the local image encoder are frozen.

[0020] Optionally, the generator includes a convolutional layer, a downsampling module, and a deep semantic editing network, wherein the deep semantic editing network includes an upsampling module, and the upsampling module includes an upsampling layer, a convolutional layer, and an affine layer.

[0021] Optionally, the aforementioned inverse generator is generated through the following steps: obtaining a set of lesion images corresponding to the aforementioned visual abnormality description; constructing an initial autoencoder network based on the aforementioned generator, wherein the encoder of the aforementioned initial autoencoder network is the aforementioned generator, the network parameters of the aforementioned encoder are fixed, and the decoder of the aforementioned initial autoencoder network is the inverse mapping of the aforementioned generator; training the aforementioned initial autoencoder network based on the aforementioned lesion image set to obtain an autoencoder network; and determining the decoder included in the aforementioned autoencoder network as the inverse generator.

[0022] Thirdly, some embodiments of this disclosure provide an electronic device, including: a display screen; one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0023] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0024] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0025] The above embodiments of this disclosure have the following beneficial effects: the image reconstruction method for visual impairment according to some embodiments of this disclosure can overcome the limitations of traditional visual aid device upgrades and iterations, and improve the experience of users with visual impairments. Specifically, the reason why device upgrades and iterations are relatively limited is that mechanical equipment updates and iterations are slow and complex. Based on this, the image reconstruction method for visual impairment according to some embodiments of this disclosure first determines the visual impairment description and lesion image corresponding to the user with visual impairment, wherein the lesion image corresponds to the visual impairment description. The lesion image can be an image observed by a person with visual impairment. Then, a pre-trained lesion image generation model corresponding to the visual impairment description is determined, wherein the lesion image generation model includes a generator, and the generator corresponds to a pre-generated inverse generator. The generator can take a normal image and the visual impairment description as input and the lesion image as output. The normal image can be an image observed by a person with normal vision. The inverse generator can be the inverse mapping of the generator. Next, a reconstructed image is generated based on the lesion image and the inverse generator. The reconstructed image can be a normal image that can be presented to a person with visual impairment. Finally, the reconstructed image is displayed. Therefore, when content needs to be presented to users with visual impairments, the generated reconstructed image can be displayed directly without the need for special vision-aiding mechanical equipment. This breaks through the limitations of traditional vision-aiding equipment upgrades and iterations, allowing users to directly view the content using devices with display functions, thus improving the experience for users with visual impairments. Attached Figure Description

[0026] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0027] Figure 1 This is an architecture diagram of an exemplary system to which some embodiments of this disclosure can be applied; Figure 2 This is a flowchart of some embodiments of the image reconstruction method for visual abnormalities according to the present disclosure; Figure 3 This is a visual diagram illustrating the feature correspondence achieved by calculating the attention maps of CLIP and DINO. Figure 4This is a schematic diagram of the pathological image generation model structure of an image reconstruction method for visual abnormalities according to some embodiments of the present disclosure; Figure 5 This is a schematic diagram of the first split structure of the DINO discriminator; Figure 6 This is a schematic diagram of the structure of the downsampling module included in the discriminator; Figure 7 This is a schematic diagram of the structure of some embodiments of the image reconstruction apparatus for visual impairment according to the present disclosure; Figure 8 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0033] Before performing any of the operations involving the collection, storage, or use of user personal information (such as descriptions of vision abnormalities) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing personal information subjects, and obtaining prior authorization and consent from personal information subjects.

[0034] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] Figure 1 An exemplary system architecture 100 for an image reconstruction method or apparatus for visual impairments, to which some embodiments of the present disclosure may be applied, is shown.

[0036] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0037] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as movie-watching applications, learning applications, guidance applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0038] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to head-mounted displays, smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0039] Server 105 can be a server that provides various services, such as training a lesion image generation model and a reverse generator, and sending the trained lesion image generation model and reverse generator to terminal devices 101, 102, and 103. Server 105 can also generate reconstructed images and send them to terminal devices 101, 102, and 103 for display.

[0040] It should be noted that the image reconstruction method for visual impairment provided in the embodiments of this disclosure can be executed by terminal devices 101, 102, and 103, or by server 105. Accordingly, the image reconstruction device for visual impairment can be located in terminal devices 101, 102, and 103, or in server 105. No specific limitations are made here.

[0041] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of an image reconstruction method for visual impairment according to the present disclosure. This image reconstruction method for visual impairment includes the following steps: Step 201: Determine the description of visual impairment and lesion images for the corresponding user with visual impairment.

[0044] In some embodiments, the entity executing the image reconstruction method for visual abnormalities (e.g.) Figure 1 The terminal device 103 shown can determine the vision abnormality description and lesion image corresponding to the user with vision abnormality. The user with vision abnormality can be a user with low vision. Low vision can refer to corrected visual acuity below a preset value, such as 0.1. The vision abnormality description can be descriptive text. For example, the vision abnormality description could be "macular degeneration, with a black spot near the central area and image distortion." The vision abnormality description can also be in English. The lesion image can be an image observed by a person with vision abnormality. The lesion image corresponds to the vision abnormality description, meaning it was generated based on the vision abnormality description. The lesion image can be from a medical eye disease database, for example, it can be an image observed by a person with vision abnormality simulated using medical methods. In practice, the vision abnormality description corresponding to the user can be pre-stored or selected or input by the user through interactive methods. The lesion image corresponding to the user and vision abnormality description can be the lesion image that needs to be reconstructed and displayed. The lesion image can be a single image or a frame from a video. The image content of the lesion image can correspond to one of the following application scenarios: teaching, watching movies, or guidance.

[0045] In some optional implementations of certain embodiments, the aforementioned executing entity may determine the description of visual impairment and lesion images of the corresponding visually impaired user through the following steps: The first step is to determine the normal image to be displayed. In practice, the normal image to be displayed can be automatically determined based on the application's logic. For example, in an educational application scenario, the normal image to be displayed could be an educational image that the current user needs to learn from.

[0046] The second step involves text encoding the aforementioned description of vision abnormalities to obtain text features. In practice, a text encoder can be used to perform text encoding on the description of vision abnormalities to obtain text features. For example, the text encoder can be a CLIP text encoder.

[0047] The third step involves inputting the normal image and the text features described above into the generator to obtain the lesion image.

[0048] In some optional implementations of certain embodiments, the above description of visual impairment can be processed by text encoding to obtain text features through the following steps: The first step is to convert the above description of vision abnormalities into the target language to obtain converted text. The target language can be English. It should be noted that if the description of vision abnormalities is already in the target language, conversion can be skipped and the text can be directly input into the text encoder to obtain text features.

[0049] The second step is to input the converted text into a text encoder to obtain text features.

[0050] Step 202: Determine the pre-trained lesion image generation model corresponding to the description of visual abnormality.

[0051] In some embodiments, the executing entity may determine a pre-trained lesion image generation model corresponding to the aforementioned vision abnormality description. This lesion image generation model may be a neural network model that takes the vision abnormality description and a normal image as input and the lesion image as output. The normal image may be an image observed by a person with normal vision. The lesion image generation model may be a generative adversarial network (GAN). For example, the lesion image generation model may be DE-NET. The lesion image generation model may be trained based on data from a medical eye disease database. The medical eye disease database may store normal images, vision abnormality descriptions, and corresponding lesion images. The lesion image generation model may include a generator. The generator corresponds to a pre-generated inverse generator. The inverse generator may be the inverse mapping of the generator. In practice, the lesion image generation model and inverse generator corresponding to the aforementioned vision abnormality description may be pre-stored. The lesion image generation model and inverse generator may be pre-trained for each vision abnormality description. The lesion image generation model and inverse generator may be trained by the executing entity or by a server.

[0052] Optionally, the aforementioned lesion image generation model may further include a discriminator. This discriminator may include a global image encoder and a local image encoder. The global and local image encoders are independent of each other within the discriminator's network structure. The global image encoder can be an image encoder with a relatively holistic and comprehensive understanding of the image description, capable of aligning different modalities such as vision and language. The local image encoder can be an image encoder that focuses on mining fine details (including shape and texture features) related to local targets, with a vision-centric approach. For example, the global image encoder could be a CLIP model (CLIP image encoder). To fully utilize the advanced knowledge encapsulated in the CLIP model to analyze complex visual scenes, multiple features can be extracted from an N-layer CLIP model (i.e., CLIP-ViT) to construct the global image encoder. The local image encoder could be a DINO model (DINO image encoder). A ViT-S model trained based on self-supervised DINO targets can be used as the backbone network of the local image encoder. The advantage of applying a self-supervised feature network is that it can encode semantic information with high quality while avoiding concerns about potential impacts on image quality metrics or inter-frame differences. The global image encoder and the local image encoder can be pre-trained. This allows for simultaneous attention to both global and local features of the image, which helps improve the accuracy of the discriminator.

[0053] Figure 3 The visualization of feature correspondences is achieved by calculating attention maps for CLIP and DINO. The evaluated features are all extracted from the same level (e.g., level 9) in both CLIP and DINO. Figure 3 It can be seen that the CLIP model can comprehensively grasp the image description, while the DINO model can uncover the fine details of local objects. It should be noted that, to better illustrate the attention biases of the CLIP and DINO models, Figure 3 The colors were preserved.

[0054] Optionally, the discriminator may further include a multi-head discriminator and a downsampling module. For example, the multi-head discriminator may be a DINO multi-head discriminator. The multi-head discriminator may take the output of the local image encoder as input. The downsampling module takes the output of the global image encoder as input. The network parameters of the global image encoder and the local image encoder are frozen.

[0055] Optionally, the generator described above may include convolutional layers, downsampling modules, and a deep semantic editing network. The deep semantic editing network may include an upsampling module. The upsampling module may include upsampling layers, convolutional layers, and affine layers. Here, the number of downsampling and upsampling modules is not limited.

[0056] As an example, the model structure of the lesion image generation model can be referenced. Figure 4 .like Figure 4 As shown, the generator takes two inputs: a normal image and a description of the vision abnormality. The vision abnormality description, after being converted to English, is fed into a large pre-trained CLIP text encoder (trained on a large number of text-image pairs) to extract sentence feature vectors, which are then converted into sentence embeddings by a learnable text encoder. This allows for greater flexibility. Normal images are encoded into a lower resolution through a series of downsampling modules. Figure 4 The lock symbol in the text indicates that the parameter is frozen.

[0057] A CLIP-guided loss function can be constructed to train the generator. Specifically, let... This represents the image features extracted after processing lesion images using a pre-trained CLIP image encoder. The CLIP-guided loss function aims to minimize... and The cosine similarity between them is expressed as: .

[0058] The goal of this loss function is to guide the editing process so that it can generate descriptions similar to the features of the input sentence. A positive CLIP-guided loss can be used to construct the generator objective for training.

[0059] Deep semantic editing networks can be built on RNN models to facilitate the global propagation of CLIP text representations within edit blocks. The initial state of a deep semantic editing network originates from image features. Specifically, it manifests as follows: MLP stands for Multilayer Perceptron Neural Network. This represents the hidden state vector at the first time step. This represents the initial value of the long-term memory unit in the LSTM at the first time step. The update rule definition for a deep semantic editing network can be as follows: .

[0060] in, Indicates the previous time step. This represents the current time step. Lines 3 to 8 of the equations listed above represent the equations for updating the standard LSTM. (Number of iterations) ( ) is the hyperparameter of Mogrifier LSTM. The original LSTM is then restored. , and These represent the forget gate, input gate, and output gate, respectively. This indicates the bias term. This represents the weight matrix. This represents element-wise matrix multiplication. This represents the sigmoid operation. This represents the hidden state vector. Based on feature extraction The initial input vector. The first two lines of the equations listed above are constructed as a multi-round interactive fusion (this operation is in...). Figure 4 (Indicated by a blue dashed line). and The parameters represent the learned parameters. Deep semantic editing networks can fully utilize the CLIP model's excellent image-text matching capabilities, thereby enhancing the semantic consistency between upsampling modules.

[0061] like Figure 4 As shown, the upsampling module includes an upsampling layer (Upsample), a convolutional layer (Conv), and two types of affine layers (C-Affine and S-Affine). Specifically, the first... The calculation process of each upsampling module can be described as follows: .

[0062] in, Indicates channel-level affine layer ( Figure 4 (C-Affine module in the text) Represents a space-level affine layer ( Figure 4 (The S-Affine module in the text). It is the updated sentence embedding generated by Mogrifier LSTM. It is a visual feature of text attention. This represents the extracted image features. yes Figure 4 The weight estimation module predicts the combined weights. The upsampling module can dynamically combine various editing modules according to different editing needs. The weight estimation module predicts the combined weights of the upsampling modules based on the inference results of the vision abnormality description and the normal image. The core idea of ​​the upsampling module is to achieve global control of the source image features in the spatial and channel dimensions, and to balance the dimensional contributions at multiple scales through an adaptive weighting mechanism.

[0063] A multi-head discriminator can be constructed using N image features extracted from ViTs. Considering that the representation dimension (token × channel) and receptive field (global) of ViTs are the same, a unified design can be adopted for all discriminator heads. The first head structure of the DINO discriminator can be as follows: Figure 5As shown, its backbone is constructed using residual convolutional modules. Finally, for the image features extracted by the DINO model and the sentence features captured by the CLIP model, we independently calculate the hinge loss for each segment, and empirically we use a four-segment structure.

[0064] The structure of the downsampling module included in the discriminator can be found in [reference]. Figure 6 Each downsampling module can contain two convolutional layers (Conv), two ReLU activation functions, and a short-circuit connection that fuses the extracted image features with the output features of the subsequent CLIP model. Figure 6 The diagram shows the structure of the first downsampling module. In practice, the discriminator can include three downsampling modules.

[0065] It should be noted that, in order to clearly indicate the structure of each part of the model and the relationships between them, Figures 4-6 The colors were preserved.

[0066] The overall objective function of the discriminator can include the adversarial loss computed by the CLIP and DINO base discriminators (i.e., using hinge loss to stabilize the adversarial training process), for example, it can be expressed as: .

[0067] in, This represents the average likelihood of the original lesion image. This represents the average likelihood of the lesion image generated by the generator. This indicates a description of visual impairment. Indicates that Incorrect descriptions of visual impairments generated based on this, for example, can be... Add noise and modify The word in the text is another word. Indicates and Related (with) The image of the lesion (for the input description of visual abnormality). Indicates and Related (with) The image of the lesion (for the input description of visual abnormality). This indicates the CLIP-based discriminator's determination of whether the input image corresponds to the input description of visual impairment. This indicates the result of the DINO-based discriminator in determining whether the input image corresponds to the input description of visual impairment. and This represents hyperparameters.

[0068] The overall objective function of the generator can be expressed as: .

[0069] in, This represents perceptual loss, used to reduce randomness in the editing process and help preserve content unrelated to the text. This indicates CLIP boot loss. and This represents two balancing weights.

[0070] Therefore, to address the challenge of limited medical eye disease image samples, a powerful discriminator can be constructed by fusing visual complementary features from a large cross-modal model within a generative adversarial network (GAN) framework. Based on this, the generator can be fine-tuned to adapt to the disease images.

[0071] In some optional implementations of certain embodiments, the above-mentioned reverse engineer may be generated through the following steps: The first step is to obtain a set of lesion images corresponding to the above description of visual abnormalities. For example, this set of lesion images can be obtained from a medical ophthalmology database.

[0072] The second step is to construct an initial autoencoder network based on the generator described above. The encoder of this initial autoencoder network is the generator described above. The network parameters of the encoder are fixed. The decoder of this initial autoencoder network is the inverse mapping of the generator described above (which can be understood as follows: if the generator is a function, then the inverse mapping of the generator is the inverse function of that function).

[0073] The third step involves training the initial autoencoder network using the aforementioned lesion image set to obtain the final autoencoder network. In practice, the lesion image set and the aforementioned visual abnormality descriptions can be used as input and output data to train the initial autoencoder network.

[0074] The fourth step is to identify the decoders included in the aforementioned autoencoder network as inverse generators.

[0075] Step 203: Generate a reconstructed image based on the lesion image and the inverse generator.

[0076] In some embodiments, the execution entity can generate a reconstructed image based on the lesion image and the inverse generator. In practice, the execution entity can input the lesion image into the inverse generator to obtain the reconstructed image. The reconstructed image can be clearly seen by a person with visual impairment. Therefore, the generated reconstructed image can be clearly seen by a user corresponding to the aforementioned description of visual impairment.

[0077] Step 204: Display the reconstructed image.

[0078] In some embodiments, the execution entity may display the reconstructed image. In practice, the reconstructed image may be displayed on the display screen of the execution entity.

[0079] In some optional implementations of certain embodiments, the reconstructed image can be displayed on a head-mounted display device corresponding to the aforementioned visually impaired user. The head-mounted display device can be a display device that can be worn on the user's head. The head-mounted display device can include, but is not limited to, AR glasses, VR glasses, and MR glasses. The head-mounted display device corresponding to the aforementioned visually impaired user can be a head-mounted display device worn by the visually impaired user. Therefore, the reconstructed image can be presented on the head-mounted display device, and the user can directly view the reconstructed image through the worn head-mounted display device.

[0080] In addressing the technical problems mentioned above using technical solutions, the following technical issues often arise in the intended application scenario: teaching or guidance. Users with visual impairments often cannot distinguish between real and algorithmic inferences in the images they see, leading to distrust (Technical Problem Two). Considering the specific requirements of teaching and guidance applications—namely, enhancing user trust in the displayed images—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the aforementioned execution entity may display the reconstructed image through the following steps: The first step is to locate the difference regions corresponding to the reconstructed image and the corresponding normal image of the lesion image, thus obtaining each difference region. In practice, the reconstructed image and the normal image can be preprocessed and aligned first. For example, image registration technology can be used to distort the reconstructed image back to the geometric space of the normal image. Then, a structural difference heatmap can be generated based on the preprocessed and aligned reconstructed image and normal image. For example, the SSIM similarity value can be calculated for each local window to obtain a similarity map. Then, the difference between 1 and each value in the similarity map can be used to obtain the structural difference heatmap. The pixel values ​​in the structural difference heatmap can be used as difference values. When the difference value of a pixel in the structural difference heatmap is close to 0, it can indicate that the structure in the local window corresponding to that pixel has hardly changed. When the difference value of a pixel in the structural difference heatmap is close to 1, it can indicate that the structure in the local window corresponding to that pixel has completely changed. Next, based on the difference values ​​of the pixels in the structural difference heatmap, the pixels in the structural difference heatmap can be clustered (e.g., K-means clustering) to obtain each difference region.

[0081] The second step is to determine the type of difference for each of the aforementioned difference regions. In practice, this can be done by determining the mean of each difference value corresponding to the difference region. Then, the difference type to which the mean belongs can be determined. Each difference type can correspond to a pre-defined range of difference values. For example, difference types can include, but are not limited to: small difference, moderate difference, and large difference.

[0082] The third step involves labeling each difference type corresponding to the aforementioned difference regions in the reconstructed image, resulting in a labeled reconstructed image. Each difference type can correspond to a pre-defined labeling method or icon. For example, the labeling method could be emitting a preset color of light at the edge of the region. The labeling icon can be a textual or icon-based hint about the difference type.

[0083] The fourth step is to display the reconstructed image after the above markings.

[0084] The above-described technical solution, as an inventive point of this disclosure, addresses technical problem two: "In teaching and guidance scenarios, visually impaired users do not know which parts of the image they see are real and which are algorithmic inferences, thus generating distrust." Factors leading to distrust of images by visually impaired users often include: they do not know which parts of the image they see are real and which are algorithmic inferences. Solving these factors can improve the trust of visually impaired users in observed images during teaching and guidance scenarios. To achieve this effect, this disclosure compares and labels the reconstructed image with a normal image, transforming the originally black-box image reconstruction process into a transparent information flow. This allows visually impaired users to entrust a portion of their visual perception to the image reconstruction algorithm, thereby increasing their trust in the observed image.

[0085] In addressing the second technical problem mentioned above, a third technical problem often arises: the accuracy of the comparison and labeling results needs improvement because the perspective of users with visual impairments is not considered. Based on the existing image of the identified lesion, we decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity can locate the difference regions corresponding to the reconstructed image based on the reconstructed image and the normal image corresponding to the lesion image through the following steps, thereby obtaining each difference region: The first step is to register the lesion image with the normal image to obtain a registered lesion image. In practice, the normal image can be used as the reference coordinate system to perform an inverse transformation on the lesion image, aligning its geometry with the normal image to obtain the registered lesion image.

[0086] The second step is to register the reconstructed image with the normal image to obtain a registered reconstructed image. In practice, the normal image can be used as a reference coordinate system to perform optical flow registration on the reconstructed image, mapping the moved object back to its original position in the normal image to obtain a registered reconstructed image.

[0087] The third step involves generating a loss map based on the registered lesion image and the normal image. Each pixel in the loss map corresponds to a degree of loss. In practice, a pixel difference map between the registered lesion image and the normal image can be generated first. This pixel difference map can be obtained by taking the absolute value of the pixel differences pixel by pixel. Then, a structural difference map between the registered lesion image and the normal image can be generated. For example, a sliding window can be used to calculate the local SSIM to generate a structural difference map with the same image size as the normal image. Next, a gradient level difference map between the registered lesion image and the normal image can be generated. This gradient level difference map can include an edge augmentation map and an edge vanishing map. For example, a gradient magnitude map between the registered lesion image and the normal image can be generated first (calculated using the Sobel operator). Then, the difference between the gradient magnitude map of the registered lesion image and the gradient magnitude map of the normal image can be taken, and the maximum value after subtracting from the difference between the gradient magnitude map of the normal image and the gradient magnitude map of the registered lesion ... registered lesion image can be taken, and the maximum value after subtracting from the difference between the gradient magnitude map of the normal image and the gradient magnitude map of the registered lesion image can be taken, and the maximum value after subtracting from the difference between the gradient magnitude map of the registered lesion image can be taken, and the maximum value after subtract Finally, the normalized pixel difference map, structural difference map, edge augmentation map, and edge vanishing map can be weighted and summed to obtain the loss map. The value of each pixel in the loss map is the weighted sum, which represents the degree of loss.

[0088] The fourth step involves generating a modification map based on the registered and reconstructed image and the normal image. Each pixel in the modification map corresponds to a degree of modification. In practice, a pixel difference map between the registered and reconstructed image and the normal image can be generated first. Then, a structural difference map between the registered and reconstructed image and the normal image can be generated. Next, a gradient level difference map between the registered and reconstructed image and the normal image can be generated. The gradient level difference map can include edge augmentation maps and edge vanishing maps. Finally, the normalized pixel difference map, structural difference map, edge augmentation map, and edge vanishing map can be weighted and summed to obtain the modification map. The value of each pixel in the modification map is the weighted sum, which represents the degree of modification. Specifically, the method for generating the modification map can refer to the specific implementation method for generating the loss map.

[0089] Fifth, based on the registered and reconstructed image and the registered lesion image, a compensation map is generated, where each pixel in the compensation map corresponds to a compensation amount. Specifically, the method for generating the compensation map can refer to the specific implementation method for generating the loss map. The value of each pixel in the compensation map is a weighted sum, which is the compensation amount.

[0090] Step 6: Based on the aforementioned loss map, modification map, and compensation map, perform semantic classification on each pixel of the reconstructed image to determine the difference type of each pixel. For each pixel in the reconstructed image, the difference type corresponding to the pixel can be determined based on the degree of loss, degree of modification, and amount of compensation in the aforementioned loss map, modification map, and compensation map. For example, numerical ranges corresponding to different numerical levels can be pre-defined for the degree of loss, degree of modification, and amount of compensation. The numerical levels can include, but are not limited to: low, high, and very high. As an example, when the degree of loss, the degree of modification, and the amount of compensation are all at a high level, the corresponding difference type can be necessary enhancement. When the degree of loss, the degree of modification, and the amount of compensation are all at a low level, the corresponding difference type can be over-processing. When the degree of loss, the degree of modification, and the amount of compensation are all at a low level, the corresponding difference type can be insufficient compensation. When the degree of loss, the degree of modification, and the amount of compensation are all at a high level, the corresponding difference type can be speculative filling. When the degree of loss, the degree of modification, and the amount of compensation are all at a low level, the corresponding difference type can be true preservation.

[0091] Step 7: Based on the determined difference types, divide the reconstructed image into different difference regions. In practice, pixels with the same difference type can be grouped into the same difference region to obtain the different difference regions.

[0092] The above-described technical solution, as an inventive point of this disclosure, addresses technical problem three: "The accuracy of comparison and labeling results needs improvement because the perspective of users with visual impairments is not considered." Factors leading to the need for improved accuracy in comparison and labeling results often include: comparing reconstructed images and normal images only allows for simple comparison of differences, failing to verify modifications to the reconstructed image from the perspective of users with visual impairments. Solving these factors improves the accuracy of comparison and labeling results. To achieve this, this disclosure introduces lesion images into the comparison. These lesion images simulate the actual view seen by users with visual impairments, providing their perspective. By comparing lesion images and normal images, a comparison can be made between the user's perspective and the real-world perspective, allowing for the identification of what the user has lost due to visual impairment. By comparing reconstructed images and normal images, a comparison can be made between the model's perspective and the real-world perspective, allowing for the identification of what modifications the model has made to the normal image. By comparing lesion images and reconstructed images, a comparison can be made between the user's perspective and the model's perspective, allowing for the identification of what losses the model has recovered. This allows for the differentiation of more specific types of differences, improving the accuracy of subsequent comparison and labeling results.

[0093] Furthermore, in the application scenario of reconstructed images for visual impairment, image comparison should focus on content differences rather than brightness differences. However, the brightness sources of diseased images, normal images, and reconstructed images are all different, and direct comparison will be subject to interference from brightness differences (e.g., misjudging difference regions, failure of SSIM calculation, and incorrect classifier thresholds). We need to ensure that the photometric baseline of the reconstructed image is consistent with that of the normal image, and the enhancement effect must be preserved. However, normalization will return the brightness to the level of the normal image, which may lead to the loss of the enhancement effect. Therefore, a hierarchical normalization solution can be adopted for reconstructed images. Ultimately, we decided to adopt the following solution: Optionally, the aforementioned implementing entity may also perform the following steps: The first step is to perform photometric normalization on the registered lesion image based on the normal image to update the registered lesion image. In practice, the histogram of the registered lesion image can be transformed into the shape of the normal image so that the overall brightness distribution of the updated registered lesion image is consistent with the normal image, but local lesion features (such as dark spots) are still preserved (because dark spots are brightness differences in spatial location, not an overall distribution problem).

[0094] The second step involves decomposing the registered and reconstructed image to obtain a base image and a detail image. In practice, edge-preserving filtering can be used to decompose the registered and reconstructed image into a base image and a detail image. For example, edge-preserving filtering can be bilateral filtering or guided filtering.

[0095] The third step is to perform photometric normalization on the base image based on the normal image to obtain a normalized base image. In practice, the histogram of the base image can be transformed into the shape of the normal image to perform photometric normalization on the base image.

[0096] The fourth step involves generating a composite image based on the normalized base image and the detail image described above, to update the registered and reconstructed image. In practice, the normalized base image and the detail image can be summed to obtain the composite image. This allows the overall luminosity of the base layer to be normalized to the reference of a normal image, while the detail layer (enhancement effect) is fully preserved. Normalizing the luminosity of both the registered and reconstructed image and the registered lesion image reduces interference from brightness differences during comparison.

[0097] The above embodiments of this disclosure have the following beneficial effects: the image reconstruction method for visual impairment according to some embodiments of this disclosure can overcome the limitations of traditional visual aid device upgrades and iterations, and improve the experience of users with visual impairments. Specifically, the reason why device upgrades and iterations are relatively limited is that mechanical equipment updates and iterations are slow and complex. Based on this, the image reconstruction method for visual impairment according to some embodiments of this disclosure first determines the visual impairment description and lesion image corresponding to the user with visual impairment, wherein the lesion image corresponds to the visual impairment description. The lesion image can be an image observed by a person with visual impairment. Then, a pre-trained lesion image generation model corresponding to the visual impairment description is determined, wherein the lesion image generation model includes a generator, and the generator corresponds to a pre-generated inverse generator. The generator can take a normal image and the visual impairment description as input and the lesion image as output. The normal image can be an image observed by a person with normal vision. The inverse generator can be a reverse mapping of the generator. Next, a reconstructed image is generated based on the lesion image and the inverse generator. The reconstructed image can be a normal image that can be presented to a person with visual impairment. Finally, the reconstructed image is displayed. Therefore, when content needs to be presented to users with visual impairments, the generated reconstructed image can be displayed directly without the need for special vision-aiding mechanical equipment. This breaks through the limitations of traditional vision-aiding equipment upgrades and iterations, allowing users to directly view the content using devices with display functions, thus improving the experience for users with visual impairments.

[0098] Further reference Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an image reconstruction apparatus for visual impairments, which are similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0099] like Figure 7 As shown, an image reconstruction apparatus 700 for visual impairment in some embodiments includes: a first determining unit 701, a second determining unit 702, a generating unit 703, and a display unit 704. The first determining unit 701 is configured to determine a description of visual impairment and a lesion image corresponding to a user with visual impairment, wherein the lesion image corresponds to the description of visual impairment. The second determining unit 702 is configured to determine a pre-trained lesion image generation model corresponding to the description of visual impairment, wherein the lesion image generation model includes a generator, and the generator corresponds to a pre-generated inverse generator. The generating unit 703 is configured to generate a reconstructed image based on the lesion image and the inverse generator. The display unit 704 is configured to display the reconstructed image.

[0100] Optionally, the display unit 704 may be further configured to display the reconstructed image in a head-mounted display device corresponding to the aforementioned visually impaired user.

[0101] Optionally, the first determining unit 701 may be further configured to: determine a normal image to be displayed; perform text encoding processing on the above-mentioned vision abnormality description to obtain text features; and input the above-mentioned normal image and the above-mentioned text features into the above-mentioned generator to obtain a lesion image.

[0102] Optionally, the first determining unit 701 may be further configured to: convert the above-mentioned vision abnormality description into the target language to obtain converted text; and input the above-mentioned converted text into a text encoder to obtain text features.

[0103] Optionally, the above-mentioned lesion image generation model further includes a discriminator, which includes a global image encoder and a local image encoder, and the global image encoder and the local image encoder are independent of each other in the network structure of the discriminator.

[0104] Optionally, the discriminator further includes a multi-head discriminator and a downsampling module. The multi-head discriminator takes the output of the local image encoder as input, and the downsampling module takes the output of the global image encoder as input. The network parameters of the global image encoder and the local image encoder are frozen.

[0105] Optionally, the generator includes a convolutional layer, a downsampling module, and a deep semantic editing network, wherein the deep semantic editing network includes an upsampling module, and the upsampling module includes an upsampling layer, a convolutional layer, and an affine layer.

[0106] Optionally, the aforementioned inverse generator is generated through the following steps: obtaining a set of lesion images corresponding to the aforementioned visual abnormality description; constructing an initial autoencoder network based on the aforementioned generator, wherein the encoder of the aforementioned initial autoencoder network is the aforementioned generator, the network parameters of the aforementioned encoder are fixed, and the decoder of the aforementioned initial autoencoder network is the inverse mapping of the aforementioned generator; training the aforementioned initial autoencoder network based on the aforementioned lesion image set to obtain an autoencoder network; and determining the decoder included in the aforementioned autoencoder network as the inverse generator.

[0107] It is understandable that the units described in the device 700 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 700 and the units contained therein, and will not be repeated here.

[0108] The following is for reference. Figure 8 It illustrates an electronic device 800 suitable for implementing some embodiments of the present disclosure (e.g., Figure 1 The diagram shows the structure of the terminal device in this disclosure. The electronic devices in some embodiments of this disclosure may include, but are not limited to, mobile terminals such as head-mounted display devices with display screens, mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs with display screens and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0109] like Figure 8 As shown, the electronic device 800 may include a processing unit 801 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0110] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 8 Each box shown can represent a device or multiple devices as needed.

[0111] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, it performs the functions defined in the methods of some embodiments of this disclosure.

[0112] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0114] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine a description of visual impairment and a lesion image corresponding to a user with visual impairment, wherein the lesion image corresponds to the description of visual impairment; determine a pre-trained lesion image generation model corresponding to the description of visual impairment, wherein the lesion image generation model includes a generator, and the generator corresponds to a pre-generated inverse generator; generate a reconstructed image based on the lesion image and the inverse generator; and display the reconstructed image.

[0115] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first determining unit, a second determining unit, a generating unit, and a display unit. The names of these units do not necessarily limit the specific unit; for example, the first determining unit may also be described as "a unit that determines a description of visual abnormality and an image of a lesion corresponding to a user with visual abnormality."

[0118] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0119] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described image reconstruction methods for visual impairment.

[0120] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An image reconstruction method for visual impairment, comprising: Determine the vision abnormality description and lesion image for the corresponding user with vision abnormality, wherein the lesion image corresponds to the vision abnormality description; A pre-trained lesion image generation model corresponding to the description of the visual abnormality is determined, wherein the lesion image generation model includes a generator and the generator corresponds to a pre-generated inverse generator; A reconstructed image is generated based on the lesion image and the inverse generator; The reconstructed image is then displayed.

2. The method according to claim 1, wherein, The process of displaying the reconstructed image includes: The reconstructed image is displayed on a head-mounted display device corresponding to the user with the visual impairment.

3. The method according to claim 1, wherein, The process of determining the description of visual abnormality and the lesion image of the corresponding user with visual abnormality includes: Identify the normal image to be displayed; The description of the vision abnormality is processed by text encoding to obtain text features; The normal image and the text features are input into the generator to obtain the lesion image.

4. The method according to claim 3, wherein, The text encoding process for the description of the visual abnormality to obtain text features includes: The description of the visual abnormality is converted into the target language to obtain the converted text; The converted text is input into a text encoder to obtain text features.

5. The method according to claim 1, wherein, The lesion image generation model also includes a discriminator, which includes a global image encoder and a local image encoder. The global image encoder and the local image encoder are independent of each other in the network structure of the discriminator.

6. The method according to claim 5, wherein, The discriminator further includes a multi-head discriminator and a downsampling module. The multi-head discriminator takes the output of the local image encoder as input, and the downsampling module takes the output of the global image encoder as input. The network parameters of the global image encoder and the local image encoder are frozen.

7. The method according to claim 1, wherein, The generator includes convolutional layers, a downsampling module, and a deep semantic editing network. The deep semantic editing network includes an upsampling module, which includes an upsampling layer, a convolutional layer, and an affine layer.

8. The method according to claim 1, wherein, The reverse engineer is generated through the following steps: Obtain the lesion image set corresponding to the description of the visual abnormality; Based on the generator, an initial autoencoder network is constructed, wherein the encoder of the initial autoencoder network is the generator, the network parameters of the encoder are fixed, and the decoder of the initial autoencoder network is the inverse mapping of the generator; Based on the lesion image set, the initial autoencoder network is trained to obtain the autoencoder network; The decoders included in the autoencoder network are identified as inverse generators.

9. An image reconstruction device for visual impairment, comprising: The first determining unit is configured to determine the vision abnormality description and lesion image of the corresponding vision abnormality user, wherein the lesion image corresponds to the vision abnormality description; The second determining unit is configured to determine a pre-trained lesion image generation model corresponding to the description of the visual abnormality, wherein the lesion image generation model includes a generator and the generator corresponds to a pre-generated inverse generator; The generation unit is configured to generate a reconstructed image based on the lesion image and the inverse generator; The display unit is configured to display the reconstructed image.

10. An electronic device, comprising: Display screen; One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images