Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a training material texture image and an actual surface topography graph corresponding to the training material texture image; processing the training material texture image through a generator in the first network model to generate a predicted surface topography graph corresponding to the training material texture image; comparing the predicted surface topography graph with the actual surface topography graph through a judger in the first network model to determine the probability that the predicted surface topography graph is the actual surface topography graph corresponding to the training material texture image; and training a judger and a generator of the first network model through the probability. According to the embodiment of the invention, the first network model can be trained based on the training material texture image and the actual surface topography graph corresponding to the training material texture image, so that the second network model can generate the corresponding surface topography graph for the material image of any material.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Haptic rendering is a tactile reproduction technology that simulates tactile feedback through multimedia devices, allowing users to perceive the tactile experience to enhance the user's immersive feeling in scenes such as virtual reality.

[0004] Currently, tactile rendering is a data-driven tactile rendering method. However, the data-driven tactile rendering method is limited by data acquisition, because obtaining real tactile information and material surface information requires the use of fine sensors and high-precision data acquisition equipment (such as tactile texture acquisition equipment), but these devices are expensive and costly to use, which greatly limits their use in large-scale research and applications.

[0005] To promote the development of tactile rendering technology, it is necessary to further reduce the cost of data collection and processing, and explore more efficient data acquisition and processing methods. Summary of the invention

[0006] The purpose of the present application is to provide an image processing method, device, electronic device and computer-readable storage medium, which can generate a real surface morphology map for a material by training the material texture image of the material through a trained neural network model.

[0007] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0008] An embodiment of the present application provides an image processing method, including: obtaining a training material texture image and an actual surface morphology map corresponding to the training material texture image; processing the training material texture image by a generator in a first network model to generate a predicted surface morphology map corresponding to the training material texture image; comparing the predicted surface morphology map with the actual surface morphology map by a judge in the first network model to determine the probability that the predicted surface morphology map is the actual surface morphology map corresponding to the training material texture image; training the judge and generator of the first network model by the probability so that the predicted surface morphology map generated by the generator corresponds to the actual surface morphology map of the training material texture image.

[0009] In some embodiments, the method also includes: obtaining a first material texture image of a first material; processing the first material texture image by the trained generator to generate a first surface morphology map corresponding to the first material texture image; obtaining a target mapping relationship between the surface morphology map and the tactile stimulation signal; determining a first tactile stimulation signal intensity corresponding to the first surface morphology map based on the target mapping relationship; and performing tactile feedback on the first object through the intensity of the first tactile stimulation signal to simulate the tactile sensation of the first object when touching the first material.

[0010] In some embodiments, according to the target mapping relationship, determining the first tactile stimulation signal intensity corresponding to the first surface morphology image includes: determining the first texture gradient curve corresponding to the first material texture image; obtaining a first mapping relationship between the texture gradient curve and the tactile perception intensity; obtaining a second mapping relationship between the tactile perception intensity and the tactile stimulation signal, the target mapping relationship being determined by the first mapping relationship and the second mapping relationship; determining the first perception intensity corresponding to the texture gradient curve through the first mapping relationship; determining the first tactile stimulation signal intensity corresponding to the first perception intensity through the second mapping relationship.

[0011] In some embodiments, obtaining the second mapping relationship between the tactile perception intensity and the tactile stimulation signal includes: performing an absolute threshold detection experiment of the tactile stimulation signal on the experimental subject to determine the minimum stimulation signal intensity that causes the tactile sensation of the experimental subject; performing a difference threshold experiment of the tactile perception signal on the experimental subject to determine the minimum stimulation signal intensity interval at which the experimental subject feels the change in stimulation intensity after causing the tactile sensation; performing a tactile stimulation amplitude experiment on the experimental subject with the minimum stimulation signal intensity as the starting value and the minimum stimulation signal intensity interval as the intensity interval to determine the tactile perception intensity corresponding to each experimental subject under different stimulation intensities; and fitting the tactile perception intensity corresponding to each experimental subject under different stimulation intensities to determine the second mapping relationship between the tactile perception intensity and the tactile stimulation signal.

[0012] In some embodiments, obtaining a first mapping relationship between a texture gradient curve and the tactile perception intensity includes: determining an actual texture gradient curve of an actual surface topography image of the training material texture image; obtaining an actual tactile perception intensity corresponding to the training material texture image; processing the actual texture gradient curve of the training material texture image through a second network model to predict a predicted tactile perception intensity corresponding to the training material texture image; training the second network model through the actual tactile perception intensity and the predicted tactile perception intensity to obtain a trained second network model, wherein the trained second network model is used to describe the first mapping relationship.

[0013] The present application provides a tactile rendering method, including: obtaining a second material texture image of a second material; determining a second surface topography map corresponding to the second material texture image; obtaining a target mapping relationship between the surface topography map and a tactile stimulation signal; determining a second tactile stimulation signal intensity corresponding to the second surface topography map according to the target mapping relationship; and performing tactile feedback on a second object through the intensity of the second tactile stimulation signal to simulate the tactile sensation when the second object touches the second material.

[0014] In some embodiments, determining a second surface morphology map corresponding to the second material texture image includes: obtaining a training material texture image and an actual surface morphology map corresponding to the training material texture image; processing the training material texture image by a generator in a first network model to generate a predicted surface morphology map corresponding to the training material texture image; comparing the predicted surface morphology map with the actual surface morphology map by a judge in the first network model to determine the probability that the predicted surface morphology map is the actual surface morphology map corresponding to the training material texture image; training the judge and generator of the first network model by the probability; and processing the second material texture image by the trained generator to generate the second surface morphology map.

[0015] An embodiment of the present application provides an image processing device, including: a training material texture image acquisition module, a predicted surface morphology image generation module, a comparison module and a training module.

[0016] Among them, the training material texture image acquisition module is used to obtain the training material texture image and the actual surface morphology map corresponding to the training material texture image; the predicted surface morphology map generation module can be used to process the training material texture image through the generator in the first network model to generate a predicted surface morphology map corresponding to the training material texture image; the comparison module can be used to compare the predicted surface morphology map with the actual surface morphology map through the judgement in the first network model to determine the probability that the predicted surface morphology map is the actual surface morphology map corresponding to the training material texture image; the training module can be used to train the judgement and generator of the first network model through the probability so that the predicted surface morphology map generated by the generator corresponds to the actual surface morphology map of the training material texture image.

[0017] The present application provides a tactile rendering device, comprising: a second material texture image acquisition module, a second surface topography map, a target mapping relationship acquisition module, a second tactile stimulation signal intensity determination module and a second tactile feedback module.

[0018] Among them, the second material texture image acquisition module can be used to acquire the second material texture image of the second material; the second surface morphology map can be used to determine the second surface morphology map corresponding to the second material texture image; the target mapping relationship acquisition module can be used to acquire the target mapping relationship between the surface morphology map and the tactile stimulation signal; the second tactile stimulation signal intensity determination module can be used to determine the second tactile stimulation signal intensity corresponding to the second surface morphology map according to the target mapping relationship; the second tactile feedback module can be used to perform tactile feedback to the second object through the second tactile stimulation signal intensity to simulate the tactile sensation when the second object touches the second material.

[0019] An embodiment of the present application provides an electronic device, which includes: a memory and a processor; the memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement any of the above-mentioned image processing methods.

[0020] An embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored to implement the image processing method as described in any one of the above items.

[0021] The embodiment of the present application provides a computer program product or a computer program, which includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the above-mentioned image processing method.

[0022] The image processing method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application can, on the one hand, train a first network model by training material texture images and actual surface morphology maps of the training material texture images, so as to generate actual surface morphology maps for any material by using the first network model and the material texture images of the material, so as to perform tactile rendering according to the surface morphology maps; on the other hand, the present application improves the probability of the surface morphology maps generated by the generator being true by continuous confrontation between the generator and the judge.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A schematic diagram of a scenario that can be applied to an image processing method or an image processing device in an embodiment of the present application is shown.

[0026] Figure 2 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0027] Figure 3 The figure is a material surface topography diagram according to an exemplary embodiment.

[0028] Figure 4 is a schematic diagram showing an image processing structure according to an exemplary embodiment.

[0029] Figure 5 The figure is a flow chart of a tactile feedback method according to an exemplary embodiment.

[0030] Figure 6 The figure is a flow chart of a method for determining the strength of a tactile stimulation signal according to an exemplary embodiment.

[0031] Figure 7 is a schematic diagram of a texture gradient curve according to an exemplary embodiment.

[0032] Figure 8 The figure is a flowchart of a method for determining a second mapping relationship according to an exemplary embodiment.

[0033] Fig. 9The figure is a flowchart of a method for determining a first mapping relationship according to an exemplary embodiment.

[0034] Fig.10 The figure is a flow chart of a tactile feedback method according to an exemplary embodiment.

[0035] Fig.11 A method for determining a second surface topography image is shown according to an exemplary embodiment.

[0036] Fig.12 It is a structural diagram of a tactile feedback method according to an exemplary embodiment.

[0037] Fig.13 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0038] Fig.14 The invention is a block diagram of a tactile rendering device according to an exemplary embodiment.

[0039] Fig.15 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0041] Those skilled in the art know that the implementation of the present application can be a system, device, equipment, method or computer program product. Therefore, the present application can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0042] Features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical solution of the present application may be put into practice and one or more of the specific details may be omitted, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present application.

[0043] The accompanying drawings are only schematic diagrams of the present application. The same reference numerals in the drawings represent the same or similar parts, and thus their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0044] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0045] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a mapping relationship that describes associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different; the terms "comprising", "including" and "having" are used to express the meaning of open inclusion and mean that in addition to the listed elements / components / etc., there may be other elements / components / etc.

[0046] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0047] The exemplary implementation scheme of the present application is described in detail below with reference to the accompanying drawings.

[0048] Figure 1 A schematic diagram of a scenario that can be applied to an image processing method or an image processing device in an embodiment of the present application is shown.

[0049] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application.

[0050] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0051] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, virtual reality devices, smart homes, etc.

[0052] The server 105 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received request data, and feed back the processing results to the terminal device.

[0053] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as basic cloud computing services such as big data and artificial intelligence platforms, etc. This application does not impose any restrictions on this.

[0054] The server 105 may, for example, obtain a training material texture image and an actual surface morphology map corresponding to the training material texture image; the server 105 may, for example, process the training material texture image through the generator in the first network model to generate a predicted surface morphology map corresponding to the training material texture image; the server 105 may, for example, compare the predicted surface morphology map with the actual surface morphology map through the judgement in the first network model to determine the probability that the predicted surface morphology map is the actual surface morphology map corresponding to the training material texture image; the server 105 may, for example, train the judgement and generator of the first network model through probability so that the predicted surface morphology map generated for the training material texture image by the trained generator is the true surface morphology map of the training material texture image.

[0055] It should be understood that Figure 1The number of terminal devices, networks and servers in the figure is only for illustration. The server 105 may be a physical server or may be composed of multiple servers. According to actual needs, it may have any number of terminal devices, networks and servers.

[0056] Those skilled in the art will know that Figure 1 The number of terminals, servers, networks and network-side devices is only illustrative, and any number of terminals, networks and servers may be provided according to actual needs. This embodiment of the present application does not limit this.

[0057] Under the above system architecture, an image processing method is provided in an embodiment of the present application, and the method can be executed by any electronic device with computing and processing capabilities.

[0058] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. The method provided in the embodiment of the present application can be executed by any electronic device with computing and processing capabilities. For example, the method can be executed by the above Figure 1 The server or terminal device in the embodiment may be executed, or the server and the terminal device may be executed together. In the following embodiments, the server is used as the execution subject for illustration, but the present application is not limited to this.

[0059] Reference Figure 2 The image processing method provided in the embodiment of the present application may include the following steps.

[0060] Step S202, obtaining a training material texture image and an actual surface topography image corresponding to the training material texture image.

[0061] In some embodiments, the actual surface topography image may be used to characterize the texture of the material surface of the training material. For example, the actual surface topography image may be used to describe the texture (or microstructure) of the material surface of the training material.

[0062] The above-mentioned material texture image may refer to the image of an object of any material, for example, it may be a texture image of a wood material (such as a certain position on a desk), a texture image of a cotton and linen material (such as a certain position on a piece of clothing), a texture image of a steel material (such as a certain position on a steel bar), a texture image of a glass material (such as a certain position on a glass water cup), etc.

[0063] In some embodiments, the material texture image may refer to a corresponding image of part or all of the texture of a certain material, and this application does not impose any limitation on this.

[0064] In some embodiments, the material texture image may be obtained by performing image acquisition (such as two-dimensional image acquisition or three-dimensional image acquisition) on an image of a certain material through an image acquisition device.

[0065] The above-mentioned training material texture image may refer to a material texture image used to train the first network model.

[0066] The first, second, etc. in the following first material texture image and second material texture image are only used for distinction and do not limit the image in this application.

[0067] The surface topography (or material surface topography) may be an image showing the texture, concavity and convexity of the material surface. The actual surface topography is used to describe the texture (such as texture or concavity and convexity structure) of the material surface of the training material.

[0068] Figure 3 is a material surface morphology diagram according to an exemplary embodiment. Figure 3 In the diagram, the horizontal and vertical axes represent two directions of a material surface, and the dark and light colors in the icon represent the microscopic height of the material in the direction perpendicular to the material surface. In some embodiments, a darker color position may represent a higher height of the position (e.g., the position is prominent), and a lighter color position may represent a lower height of the position (e.g., the position is relatively flat).

[0069] In related technologies, a morphology measurement and analysis device (Surface Morphology Analyzer) can be used to perform high-resolution scanning on the surface of a material (or material) to obtain images of the material surface. These images can show the texture, concavity and convexity of the material surface, thereby obtaining a surface morphology map of the material.

[0070] In some embodiments, the actual surface morphology image may refer to an image obtained by actually measuring a material object through a device (such as the above-mentioned morphological measurement and analysis device) and capable of describing the texture, concavity, convexity, shape, etc. of the material.

[0071] Step S204: Process the training material texture image through the generator in the first network model to generate a predicted surface morphology image corresponding to the training material texture image.

[0072] In some embodiments, the first network model may include a generator (e.g. Figure 4 Generator 401 in Figure 4 Determinator 402 in ).

[0073] In some embodiments, the generator (e.g. Figure 4The generator 401 in the example may generate a training material texture image (such as Figure 4 The material texture image 403 in the training material texture image is processed to generate a predicted surface morphology image (such as Figure 4 Surface topography image 404 generated in .

[0074] Step S206, comparing the predicted surface topography image with the actual surface topography image through the judgement device in the first network model to determine the probability that the predicted surface topography image is the actual surface topography image corresponding to the training material texture image.

[0075] The above discriminator (such as Figure 4 The discriminator 402 in the example will respectively analyze the actual surface topography (such as Figure 4 The real surface morphology map 405 in the generator and the fake predicted surface morphology map generated by the generator (such as Figure 4 The generated surface morphology map 404 in the example is evaluated and a probability value is output, which can be used to indicate the probability that the predicted surface morphology map is the true surface morphology map of the training material.

[0076] Step S208, training the judger and the generator of the first network model by probability, so that the predicted surface morphology map generated by the trained generator for the training material texture image is the real surface morphology map of the training material texture image.

[0077] In some embodiments, the distance difference between the predicted surface morphology image and the actual surface morphology image can be calculated, and then a loss function can be determined based on the distance difference. The judger and the generator of the first network model are trained by the loss function so that the judger cannot distinguish the predicted surface morphology image generated by the generator from the actual surface morphology image.

[0078] The goal of the above generator is to deceive the discriminator so that it cannot distinguish whether the generated material surface morphology is real or fake. Therefore, the loss function of the generator needs to maximize the probability value output by the discriminator, so that the generated surface morphology is closer to the actual surface morphology.

[0079] The goal of the discriminator is to distinguish the actual surface topography from the predicted surface topography as much as possible, and output the probability value of the predicted surface topography being the real surface topography. Therefore, the loss function of the discriminator needs to minimize the probability value of the fake surface topography generated by the generator, so that the discriminator can better distinguish between real and fake surface topography.

[0080] By continuously iterating the training of the generator and the discriminator, the generator can generate fake surface topography that is closer and closer to the actual surface topography, and the discriminator will become more accurate in distinguishing between the real surface topography and the fake surface topography.

[0081] Finally, after many iterations of training, the generator can generate very realistic fake surface morphology maps, thus achieving the goal of generating new data similar to real data and achieving the goal of autonomously generating high-quality material surface morphology maps.

[0082] Through the above-mentioned trained network, a corresponding surface morphology map can be generated for the material texture image of any material.

[0083] That is, through the above method, when you want to know the surface morphology of a certain material surface exactly, you do not need to use precision instruments to scan the material surface. You only need to collect the material texture image corresponding to the material surface through a two-dimensional image (or three-dimensional image) acquisition device, and then use the generator trained in this embodiment to process it, so as to obtain the surface morphology corresponding to the material.

[0084] The technical solution provided by the above embodiment can train the first network model based on the training material texture image and the actual surface morphology map corresponding to the training material texture image, so that the second network model can generate a corresponding surface morphology map for any material texture image.

[0085] Through the above method, a material surface morphology map can be automatically given for any material, so as to create a large tactile image database for tactile feedback training and tactile feedback operations.

[0086] In some embodiments, after obtaining a surface morphology map of a material (which may be an actual surface morphology map or a surface morphology map predicted and generated by a generator), a target mapping relationship between the surface morphology map and the tactile stimulation signal may be obtained; based on the target mapping relationship, the intensity of the tactile stimulation signal corresponding to the surface morphology map may be determined; and tactile feedback of the tactile stimulation signal intensity may be used to simulate the tactile sense of the material texture.

[0087] Next, this application will be Figure 5 The illustrated embodiment explains the above-mentioned tactile simulation process.

[0088] Figure 5 The figure is a flow chart of a tactile feedback method according to an exemplary embodiment.

[0089] refer to Figure 5 The above-mentioned tactile feedback method may include the following steps.

[0090] Step S502: Acquire a first material texture image of a first material.

[0091] The above-mentioned first material can refer to any material such as wood, iron, glass, plastic, etc., and this application does not impose any limitation on this.

[0092] The first material texture image may refer to an image obtained by capturing a portion of the surface or the entire surface of an object corresponding to the first material.

[0093] Step S504: Process the first material texture image through the trained generator to generate a first surface topography image corresponding to the first material texture image.

[0094] In some embodiments, Figure 2 In the embodiment, the trained generator processes the first material texture image to generate a first surface morphology map corresponding to the first material texture image. The first surface morphology map may refer to a surface morphology map that can describe the texture, convexity and shape of the first material in the first material texture image.

[0095] Step S506, obtaining a target mapping relationship between the surface topography image and the tactile stimulation signal.

[0096] The tactile stimulation signal may refer to a signal that can stimulate the tactile feedback device so that the tactile feedback device generates tactile feedback to the user. The tactile stimulation signal may refer to any signal that can act on the tactile feedback device so that the tactile feedback device generates tactile feedback, such as direct current, alternating current, direct voltage, alternating current power, etc.

[0097] In some embodiments, a target mapping relationship between a surface topography image and a tactile stimulation signal can be established in advance, and the target mapping relationship can correspond the surface topography image to the tactile stimulation signal one by one. That is, as long as a material surface topography image corresponding to a material texture image is given, a corresponding tactile stimulation signal can be uniquely determined through the target mapping relationship.

[0098] In some embodiments, the above-mentioned target mapping relationship can be described by one, two or more functional relationships, and can also be described by a neural network, etc., and the present application does not impose any restrictions on this.

[0099] Step S508: determining the first tactile stimulation signal intensity corresponding to the first surface topography image according to the target mapping relationship.

[0100] In some embodiments, the tactile stimulation signal may be divided into multiple intensities, and different intensities may prompt the tactile feedback device to generate tactile feedback of different intensities to the target user.

[0101] In some embodiments, the tactile stimulation signal intensity may refer to the voltage amplitude of a DC voltage, the current amplitude of a DC current, the voltage frequency or amplitude of an AC voltage, the current frequency or amplitude of an AC current, etc.

[0102] In some embodiments, the surface topography of different materials may cause different tactile sensations to the user, and thus the tactile stimulation signal intensities corresponding to the surface topography of different materials may also be different.

[0103] Step S510 , performing tactile feedback on the first object through the intensity of the first tactile stimulation signal to simulate the tactile sensation when the first object touches the first material.

[0104] In some embodiments, the first tactile stimulation signal intensity may be applied to a tactile stimulation device (also referred to as a tactile reproduction device) to provide tactile feedback to the first object.

[0105] The tactile stimulation device can increase or decrease the pressure, temperature, humidity, etc. of the first object so that the user can feel different tactile stimulations.

[0106] Tactile reproduction devices generally use vibration, tapping, electric current, heat and cold and other stimuli that can cause tactile receptors in the skin to respond to reproduce the touch-related sensations such as touch, pressure, slip, heat and cold, and pain generated during the interaction.

[0107] It is understandable that the present application does not limit the form of tactile feedback provided by the tactile stimulation device (or tactile reproduction device) to the user, such as tactile feedback through mechanical vibration, tactile feedback through static electricity, etc.

[0108] The technical solution provided by the above embodiment can uniquely determine a first tactile stimulation signal intensity through a target mapping relationship of a first surface topography image of a first material, so as to provide tactile feedback to a human body through the first tactile stimulation signal intensity to accurately simulate the tactile sensation when a first object touches the first material.

[0109] In some embodiments, determining the first tactile stimulation signal intensity corresponding to the surface topography according to the target mapping relationship may include: determining a first texture gradient curve corresponding to the material texture image of the surface topography; obtaining a first mapping relationship between the texture gradient curve and the tactile perception intensity; obtaining a second mapping relationship between the tactile perception intensity and the tactile stimulation signal, so as to determine the first tactile stimulation signal intensity corresponding to the surface topography according to the first texture gradient curve, the first mapping relationship, and the second mapping relationship. Wherein, determining the first tactile stimulation signal intensity corresponding to the surface topography according to the first texture gradient curve, the first mapping relationship, and the second mapping relationship includes: determining the first perception intensity corresponding to the texture gradient curve through the first mapping relationship; and determining the first tactile stimulation signal intensity corresponding to the first perception intensity through the second mapping relationship.

[0110] Below, this application will be Figure 6 The above process is explained as an example.

[0111] Figure 6 The figure is a flow chart of a method for determining the strength of a tactile stimulation signal according to an exemplary embodiment.

[0112] refer to Figure 6 The above-mentioned method for determining the strength of the tactile stimulation signal can be performed by referring to the following steps.

[0113] Step S602, determining a first texture gradient curve corresponding to the surface topography image.

[0114] A texture gradient curve is a curve graph used to describe the texture changes in an object or scene.

[0115] The texture gradient curve mentioned in this application is an image that uses a cross-sectional curve to show the height variation of the material surface through which the dotted line passes.

[0116] The texture gradient curve is a curve used to describe the degree of texture change on the surface of a material. This curve reflects the trend of the texture change on the surface of the material at different positions. The texture gradient curve usually shows the change of the texture density, direction, frequency, etc. on the surface of the material with the position. The shape and trend of the curve can reflect the roughness of the material surface, the continuity and regularity of the texture change. For example, the steepness of the curve indicates the drastic degree of texture change, and the periodicity of the curve reflects the repeatability of the texture.

[0117] In some embodiments, if Figure 3 If we cut a cross section from the dotted line in the surface morphology shown in the figure, we can get the following Figure 7 The texture gradient curve is shown. Figure 7 The texture gradient curve shown can be Figure 3 The roughness at the middle dotted line reflects the Figure 7 The position with larger gradient represents Figure 3 The darker the color in the image, the higher the height at that location. If the texture gradient curve has a small change period and a large change amplitude, it can be determined that the material surface corresponding to the texture gradient curve has a strong sense of concavity and convexity, which is more irritating to the human body; if the texture gradient curve has a large change period or a small change amplitude, it can be determined that the material surface corresponding to the texture gradient curve is relatively smooth, which is less irritating to the human body.

[0118] Step S604: acquiring a first mapping relationship between the texture gradient curve and the tactile perception intensity.

[0119] The above-mentioned tactile perception intensity can be the level of tactile perception set by those skilled in the art according to actual needs. For example, the just-perceived touch can be defined as level 1 tactile perception intensity, the tingling feeling can be defined as level 2 tactile perception intensity, the strong tingling can be defined as level 3 tactile perception intensity, and so on. The present application does not limit the definition of tactile perception intensity.

[0120] In some embodiments, the first mapping relationship may map the texture gradient curve to the tactile perception intensity one by one, that is, different texture gradient curves may correspond to different tactile perception intensities, for example, a texture gradient curve with a small change period or a large change amplitude may correspond to a large tactile perception intensity; for another example, a texture gradient curve with a large change period or a small change amplitude may correspond to a small tactile perception intensity, etc.

[0121] Step S606: acquiring a second mapping relationship between the tactile perception intensity and the tactile stimulation signal, wherein the target mapping relationship is determined by the first mapping relationship and the second mapping relationship.

[0122] In some embodiments, the second mapping relationship may describe a one-to-one mapping relationship between tactile perception intensity and tactile stimulation signal.

[0123] Step S608: determining a first perception intensity corresponding to the texture gradient curve through a first mapping relationship.

[0124] Step S610: determining the first tactile stimulation signal intensity corresponding to the first perception intensity through the second mapping relationship.

[0125] Then, through the above-mentioned first mapping relationship and the second mapping relationship, a direct one-to-one mapping relationship between the surface morphology image and the tactile stimulation signal can be described, that is, through the first mapping relationship and the second mapping relationship, a corresponding tactile stimulation signal can be determined for the surface morphology image, so as to provide tactile feedback to the target object through the unique tactile stimulation signal to simulate the tactile sensation when the first object touches the first material.

[0126] Through the above method, the surface morphology map and the tactile perception signal can be matched one by one through the first mapping relationship and the second mapping relationship, so as to accurately determine the corresponding first tactile stimulation signal intensity for the texture gradient curve of the first material, so as to provide tactile feedback to the first object through the first tactile stimulation signal intensity, so as to better simulate the tactile feeling of the first object touching the first material.

[0127] Figure 8 The figure is a flowchart of a method for determining a second mapping relationship according to an exemplary embodiment.

[0128] refer to Figure 8, the above-mentioned second mapping relationship determination method may include the following steps.

[0129] Step S802: Perform an absolute threshold detection experiment of the tactile stimulation signal on the experimental subject to determine the minimum stimulation signal intensity that causes the tactile sensation of the experimental subject.

[0130] The above-mentioned experimental subject may refer to a person or an animal participating in the experiment or other objects capable of tactile perception, and there may be multiple experimental subjects.

[0131] In some embodiments, the tactile perception threshold may include a tactile perception absolute threshold and a resolution threshold, wherein the tactile perception absolute threshold may refer to the minimum stimulus signal that can just induce tactile sensation, and the tactile perception resolution threshold may refer to the minimum signal interval at which the experimental subject can feel the change in stimulus intensity after induced tactile sensation.

[0132] The above-mentioned tactile perception absolute threshold (or tactile perception discrimination threshold) can be determined by an average value, a median value, a maximum value or a minimum value algorithm.

[0133] The above-mentioned absolute threshold detection experiment may refer to performing tactile stimulation on each experimental subject to determine the minimum stimulation signal that each experimental subject can feel the tactile sensation, and then averaging (taking the median, maximum value, or minimum value, etc.) the minimum stimulation signal to determine the minimum stimulation signal intensity that causes the experimental subject's tactile sensation.

[0134] Step S804: Perform a difference threshold test of tactile perception signals on the experimental subject to determine the minimum stimulation signal intensity interval at which the experimental subject feels a change in stimulation intensity after induced tactile sensation.

[0135] The difference threshold experiment may refer to changing the tactile stimulation for each experimental subject to determine the minimum signal interval at which the experimental subject can feel the change in stimulation intensity after the tactile sensation is induced, thereby determining the minimum stimulation signal intensity interval at which the experimental subject can feel the change in stimulation intensity.

[0136] Step S806 , performing a tactile stimulation amplitude experiment on the experimental subjects with the minimum stimulation signal intensity as the starting value and the minimum stimulation signal intensity interval as the intensity interval, so as to determine the tactile perception intensity corresponding to each experimental subject under different stimulation intensities.

[0137] In some embodiments, after the minimum stimulation signal and the minimum stimulation signal intensity interval are determined through the experimental subjects, a tactile stimulation amplitude experiment can be conducted on the experimental subjects again with the minimum stimulation signal intensity as the starting value and the minimum stimulation signal intensity interval as the intensity interval to determine the tactile perception intensity corresponding to each experimental subject under different stimulation intensities.

[0138] Step S808 , fitting the tactile perception intensity corresponding to each experimental object under different stimulation intensities to determine a second mapping relationship between the tactile perception intensity and the tactile stimulation signal.

[0139] Through the technical solution provided by the above embodiment, the second mapping relationship between the tactile perception intensity and the tactile stimulation signal can be determined.

[0140] Fig. 9 The figure is a flowchart of a method for determining a first mapping relationship according to an exemplary embodiment.

[0141] refer to Fig. 9 , the above-mentioned first mapping relationship determination method may include the following steps.

[0142] Step S902, determining an actual texture gradient curve of an actual surface topography image of a training material texture image.

[0143] The device can be used to scan the surface of the object corresponding to the training material to obtain an actual texture surface morphology map that can describe the texture, bumps and shapes of the material; then obtain a cross-section of the actual texture surface morphology map, and finally determine the actual texture gradient curve corresponding to the actual surface morphology map based on the cross-section.

[0144] The above-mentioned actual surface morphology image may refer to a surface morphology image actually acquired through relevant equipment.

[0145] Step S904, obtaining the actual tactile perception intensity corresponding to the training material texture image.

[0146] The actual tactile perception intensity mentioned above may refer to the tactile perception intensity actually felt by the target object when it actually touches the training material corresponding to the training material texture image.

[0147] Step S906: Processing the actual texture gradient curve of the training material texture image through the second network model to predict the predicted tactile perception intensity corresponding to the training material texture image.

[0148] The above-mentioned second network model can be a neural network model, a machine learning network model, etc., and this application does not impose any restrictions on this.

[0149] Step S908: training the second network model by using the actual tactile perception intensity and the predicted tactile perception intensity to obtain a trained second network model, wherein the trained second network model is used to describe the first mapping relationship.

[0150] In some other embodiments, the experimental subject can touch different material surfaces to determine the tactile perception intensity of the experimental subject when touching the material surface, thereby determining the corresponding relationship between the surface topography corresponding to the material surface and the tactile perception intensity. Then, through the above experiment, the one-to-one mapping relationship (i.e., the first mapping relationship) between different material surface topography and tactile perception intensity can be determined.

[0151] It is understandable that the present application does not limit the method for determining the first mapping relationship.

[0152] Through the above method, the direct mapping relationship between the tactile perception intensity and the texture gradient curve can be accurately determined.

[0153] Fig.10 The figure is a flow chart of a tactile feedback method according to an exemplary embodiment.

[0154] The above tactile feedback method may include the following steps.

[0155] Step S1002, obtaining a second material texture image of a second material.

[0156] The above-mentioned second material can refer to any material such as wood, iron, glass, plastic, etc., and this application does not impose any limitation on this.

[0157] The second material texture image may refer to an image obtained by capturing a portion of the surface or the entire surface of an object corresponding to the second material.

[0158] Step S1004: determine a second surface topography image corresponding to the second material texture image.

[0159] Step S1006, obtaining a target mapping relationship between the surface topography image and the tactile stimulation signal.

[0160] Step S1008 , determining the second tactile stimulation signal intensity corresponding to the second surface topography according to the target mapping relationship.

[0161] Step S1010 , performing tactile feedback on the second object through the intensity of the second tactile stimulation signal to simulate the tactile sensation when the second object touches the second material.

[0162] Through the above method, the second surface morphology map corresponding to the second material can be determined according to the second material texture image of the second material, and then the second tactile stimulation signal intensity corresponding to the second surface morphology map can be determined according to the target mapping relationship between the surface morphology map and the tactile stimulation signal, and then the second tactile stimulation signal intensity can be used to perform tactile feedback on the second object to simulate the tactile feeling of the second object when touching the second material.

[0163] The above method does not require precision equipment to collect prior data such as material surface morphology, tactile perception intensity, and tactile perception signals. It can determine the tactile stimulation signal intensity corresponding to a certain material based on the surface image of the material, and then provide feedback to the user through the determined tactile stimulation signal intensity, thereby simulating the tactile sensation of the user touching the surface of the material.

[0164] In some embodiments, in the VR field of artificial intelligence, the most important research topic at present is tactile research. The present application can be applied to the VR field to provide tactile feedback to VR users through VR devices.

[0165] For example, a VR device can be used to extract a material surface image of an object that a user is about to touch to obtain a material surface image of the object; then, the material surface image is processed using a generator in the first network model provided in the present application to determine a surface morphology map of the material corresponding to the object; then, a target mapping relationship between the surface morphology map and the tactile stimulation signal is obtained; then, based on the target mapping relationship, the tactile stimulation signal intensity corresponding to the surface morphology map of the material of the object is determined; finally, the determined tactile stimulation signal intensity is used to stimulate the VR device to provide tactile feedback to the user through the VR device, thereby simulating the tactile feeling of the user touching the material.

[0166] The above method can accurately and quickly determine the tactile stimulation signal strength corresponding to the material of the touch object when the surface image of the touch object is obtained, so as to provide tactile feedback to the user through the determined tactile stimulation signal strength, thereby better simulating the tactile feeling of the user touching the material. The above process is simple and convenient, and does not require the collection of prior data, which greatly saves manpower and material resources.

[0167] Fig.11 A method for determining a second surface topography image is shown according to an exemplary embodiment.

[0168] refer to Fig.11 , the above-mentioned second surface morphology determination method may include the following steps.

[0169] Step S1102, obtaining a training material texture image and an actual surface topography image corresponding to the training material texture image.

[0170] Step S1104: Process the training material texture image through the generator in the first network model to generate a predicted surface morphology image corresponding to the training material texture image.

[0171] Step S1106, comparing the predicted surface topography image with the actual surface topography image through the judgement device in the first network model to determine the probability that the predicted surface topography image is the actual surface topography image corresponding to the training material texture image.

[0172] Step S1108, training the judger and the generator of the first network model by probability.

[0173] Step S1110, processing the second material texture image by the trained generator to generate a second surface topography image.

[0174] Through the above method, the corresponding material morphology map can be accurately and conveniently determined for different materials without the need for precision instruments.

[0175] Fig.12 It is a structural diagram of a tactile feedback method according to an exemplary embodiment.

[0176] like Fig.12 As shown, the tactile feedback method may correspond to the following modules: a tactile perception stimulation amplitude module 1201 , an autonomous texture gradient curve generation module 1202 , and a tactile rendering module 1203 .

[0177] The functions and execution flow of each module are as follows:

[0178] 1. Tactile perception stimulus amplitude module: conduct absolute threshold and resolution threshold experiments to provide a theoretical basis for the selection of parameters for tactile reproduction rendering methods.

[0179] An experiment on tactile perception stimulation amplitude was conducted, and it was found that the relationship between stimulation intensity and perception intensity is close to a power function (i.e., the second mapping relationship), that is, there is a positive correlation between stimulation intensity and perception intensity.

[0180] 2. Autonomously generate texture gradient curve module: When a new material image is input and tactile rendering is required, the first network model trained by the above embodiment is used to autonomously generate the surface morphology map of the tactile material. In the process of autonomously generating the material surface morphology map, the new material texture image is used as the input sample of the first network model to generate the corresponding material surface morphology map, and the corresponding real material surface morphology map is used as the real data sample. The data samples generated by the generator and the real data samples are input into the discriminator for true and false judgment, and after multiple generations of generation by the generator and true and false identification by the discriminator, the adversarial model reaches the target error.

[0181] 3. Tactile rendering module: According to the first mapping system between the material texture gradient curve and the human tactile perception, and the second mapping relationship between the human tactile perception and the stimulation amplitude, the stimulation amplitude of the stimulation signal corresponding to the input image is calculated, the tactile stimulation signal is generated, and the tactile rendering signal is input into the tactile rendering device to complete the tactile rendering of the corresponding material.

[0182] The above method has the following beneficial effects.

[0183] 1. This application uses the first network model to autonomously generate material surface morphology maps, avoiding the work of manually establishing a tactile texture database, thereby reducing labor costs such as manual collection, annotation and storage of data, reducing a lot of time and resources, and effectively reducing the cost of establishing a tactile image database.

[0184] 2. The method proposed in this application can avoid the problem of not being able to generate tactile stimulation signals due to the new material not existing in the database, that is, the tactile rendering function can be realized and will no longer be limited by whether the material-related images and surface morphology data already exist in the database.

[0185] 3. The method for autonomously generating image data proposed in this application can generate high-quality material surface morphology images, avoiding the problem of poor quality of tactile material related data collected due to problems with the tactile texture acquisition equipment in the original solution.

[0186] 4. The method for autonomously generating material surface topography maps proposed in this application can create an ultra-large tactile texture image database with low resource consumption, which can be used for the training and research of large-scale tactile rendering models.

[0187] The present application proposes a method for autonomously generating a tactile surface topography map of a material surface. It solves the problems of limited data collection methods, limited collection quantity, and high manual annotation costs in the existing technical solutions for creating a large tactile image database. Based on the original solution, the first network model is used to autonomously generate a surface topography map of a tactile material; the present application can also obtain a texture gradient curve through a surface topography profile, and then calculate the stimulus signal amplitude through the relationship that the texture gradient curve is proportional to the human tactile perception (first mapping relationship) and the human tactile perception is positively correlated with the stimulus signal (second mapping relationship), and input it into a tactile rendering device. Therefore, when a new material texture needs to be rendered, there is no need to worry about whether the material texture gradient curve already exists in the database. The material texture gradient curve can be directly obtained through this solution, and then the corresponding tactile stimulation signal is generated. This solution greatly reduces the data collection cost and manual annotation cost of the tactile database, so that the establishment of a large tactile database is no longer restricted by collection and annotation, which is conducive to promoting the establishment of a large tactile rendering training model.

[0188] It should be particularly pointed out that each step in each embodiment of the above-mentioned image processing method can be mutually crossed, replaced, added, or deleted. Therefore, these reasonable permutations, combinations, and transformations of the image processing method should also fall within the protection scope of this application, and the protection scope of this application should not be limited to the embodiments.

[0189] Based on the same inventive concept, the present application also provides an image processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0190] Fig.13 is a block diagram of an image processing device according to an exemplary embodiment. Fig.13 The image processing device 1300 provided in the embodiment of the present application may include: a training material texture image acquisition module 1301, a predicted surface morphology map generation module 1302, a comparison module 1303 and a training module 1304.

[0191] Among them, the training material texture image acquisition module 1301 can be used to obtain the training material texture image and the actual surface morphology map corresponding to the training material texture image; the predicted surface morphology map generation module 1302 can be used to process the training material texture image through the generator in the first network model to generate a predicted surface morphology map corresponding to the training material texture image; the comparison module 1303 can be used to compare the predicted surface morphology map with the actual surface morphology map through the judgement in the first network model to determine the probability that the predicted surface morphology map is the actual surface morphology map corresponding to the training material texture image; the training module 1304 can be used to train the judgement and generator of the first network model through probability, so that the predicted surface morphology map generated by the trained generator for the training material texture image is the true surface morphology map of the training material texture image.

[0192] It should be noted that the training material texture image acquisition module 1301, the predicted surface topography image generation module 1302, the comparison module 1303 and the training module 1304 correspond to S202-208 in the method embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content applied for in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0193] In some embodiments, the image processing device 1300 may further include: a first material texture image acquisition module, a first surface topography image determination module, a target mapping relationship acquisition module, a first tactile stimulation signal intensity determination module, and a first tactile feedback module.

[0194] Among them, the first material texture image acquisition module can be used to acquire the first material texture image of the first material; the first surface morphology map determination module can be used to process the first material texture image through a trained generator to generate a first surface morphology map corresponding to the first material texture image; the target mapping relationship acquisition module can be used to acquire the target mapping relationship between the surface morphology map and the tactile stimulation signal; the first tactile stimulation signal intensity determination module can be used to determine the first tactile stimulation signal intensity corresponding to the first surface morphology map according to the target mapping relationship; the first tactile feedback module can be used to perform tactile feedback on the first object through the first tactile stimulation signal intensity to simulate the tactile sensation when the first object touches the first material.

[0195] In some embodiments, the first tactile stimulation signal strength determination module may include: a first texture gradient curve determination submodule, a first mapping relationship acquisition submodule, a second mapping relationship acquisition submodule, a first perception strength determination submodule and a first tactile stimulation signal strength determination submodule.

[0196] Among them, the first texture gradient curve determination submodule can be used to determine the first texture gradient curve corresponding to the first material texture image; the first mapping relationship acquisition submodule can be used to obtain the first mapping relationship between the texture gradient curve and the tactile perception intensity; the second mapping relationship acquisition submodule can be used to obtain the second mapping relationship between the tactile perception intensity and the tactile stimulation signal, and the target mapping relationship is determined by the first mapping relationship and the second mapping relationship; the first perception intensity determination submodule can be used to determine the first perception intensity corresponding to the texture gradient curve through the first mapping relationship; the first tactile stimulation signal intensity determination submodule can be used to determine the first tactile stimulation signal intensity corresponding to the first perception intensity through the second mapping relationship.

[0197] In some embodiments, the second mapping relationship acquisition submodule may include: an absolute threshold detection experiment unit, a difference threshold experiment unit, a tactile stimulation amplitude experiment unit, and a fitting unit.

[0198] Among them, the absolute threshold detection experiment unit can be used to conduct an absolute threshold detection experiment of tactile stimulation signals on the experimental subjects to determine the minimum stimulation signal intensity that causes tactile sensation in the experimental subjects; the difference threshold experiment unit can be used to conduct a difference threshold experiment of tactile perception signals on the experimental subjects to determine the minimum stimulation signal intensity interval at which the experimental subjects feel the change in stimulation intensity after causing the tactile sensation; the tactile stimulation amplitude experiment unit can be used to conduct a tactile stimulation amplitude experiment on the experimental subjects with the minimum stimulation signal intensity as the starting value and the minimum stimulation signal intensity interval as the intensity interval to determine the tactile perception intensity corresponding to each experimental subject under different stimulation intensities; the fitting unit can be used to fit the tactile perception intensity corresponding to each experimental subject under different stimulation intensities to determine the second mapping relationship between the tactile perception intensity and the tactile stimulation signal.

[0199] In some embodiments, the first mapping relationship acquisition submodule may include: an actual texture gradient curve acquisition unit, an actual tactile perception intensity acquisition unit, a predicted tactile perception intensity prediction unit, and a second network model training unit.

[0200] Among them, the actual texture gradient curve acquisition unit can be used to determine the actual texture gradient curve of the actual surface morphology map of the training material texture image; the actual tactile perception intensity acquisition unit can be used to obtain the actual tactile perception intensity corresponding to the training material texture image; the predicted tactile perception intensity prediction unit can be used to process the actual texture gradient curve of the training material texture image through the second network model to predict the predicted tactile perception intensity corresponding to the training material texture image; the second network model training unit can be used to train the second network model through the actual tactile perception intensity and the predicted tactile perception intensity to obtain a trained second network model, wherein the trained second network model is used to describe the first mapping relationship.

[0201] Since the functions of the device 1300 have been described in detail in the corresponding method embodiments, this application will not repeat them here.

[0202] Fig.14 is a block diagram of a tactile rendering device according to an exemplary embodiment. Fig.14 The tactile rendering device 1400 provided in the embodiment of the present application may include: a second material texture image acquisition module 1401, a second surface topography determination module 1402, a target mapping relationship acquisition module 1403, a second tactile stimulation signal intensity determination module 1404 and a second tactile feedback module 1405.

[0203] Among them, the second material texture image acquisition module 1401 can be used to acquire the second material texture image of the second material; the second surface morphology map acquisition module 1402 can be used to determine the second surface morphology map corresponding to the second material texture image; the target mapping relationship acquisition module 1403 can be used to acquire the target mapping relationship between the surface morphology map and the tactile stimulation signal; the second tactile stimulation signal intensity determination module 1404 can be used to determine the second tactile stimulation signal intensity corresponding to the second surface morphology map according to the target mapping relationship; the second tactile feedback module 1405 can be used to perform tactile feedback to the second object through the second tactile stimulation signal intensity to simulate the tactile sensation when the second object touches the second material.

[0204] In some embodiments, the second surface morphology map acquisition module 1402 may include: a second acquisition submodule of an actual surface morphology map, a predicted surface morphology map generation submodule corresponding to a training material texture image, a comparison submodule, a first network model training submodule, and a second surface morphology map generation submodule.

[0205] Among them, the second acquisition submodule of the actual surface morphology map can be used to obtain the training material texture image and the actual surface morphology map corresponding to the training material texture image; the predicted surface morphology map generation submodule corresponding to the training material texture image can be used to process the training material texture image through the generator in the first network model to generate the predicted surface morphology map corresponding to the training material texture image; the comparison submodule can be used to compare the predicted surface morphology map with the actual surface morphology map through the judgement in the first network model to determine the probability that the predicted surface morphology map is the actual surface morphology map corresponding to the training material texture image; the first network model training submodule can be used to train the judgement and generator of the first network model by probability; the second surface morphology map generation submodule can be used to process the second material texture image through the trained generator to generate a second surface morphology map.

[0206] Since the functions of the device 1400 have been described in detail in the corresponding method embodiments, this application will not repeat them here.

[0207] The modules and / or submodules and / or units described in the embodiments of the present application may be implemented in software or hardware. The modules and / or submodules and / or units described may also be arranged in a processor. The names of these modules and / or submodules and / or units do not, in certain circumstances, constitute limitations on the modules and / or submodules and / or units themselves.

[0208] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module or program segment, and a part of the above-mentioned module or program segment contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer program instructions.

[0209] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0210] Fig.15 The structure diagram of the electronic device suitable for implementing the embodiment of the present application is shown. It should be noted that: Fig.15 The electronic device 1500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0211] like Fig.15 As shown, the electronic device 1500 includes a central processing unit (CPU) 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage part 1508 into a random access memory (RAM) 1503. In the RAM 1503, various programs and data required for the operation of the electronic device 1500 are also stored. The CPU 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0212] The following components are connected to the I / O interface 1505: an input section 1506 including a keyboard, a mouse, etc.; an output section 1507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1508 including a hard disk, etc.; and a communication section 1509 including a network interface card such as a LAN card, a modem, etc. The communication section 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to the I / O interface 1505 as needed. A removable medium 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1510 as needed, so that a computer program read therefrom is installed into the storage section 1508 as needed.

[0213] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes computer program instructions for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1509, and / or installed from a removable medium 1511. When the computer program is executed by a central processing unit (CPU) 1501, the above-mentioned functions defined in the system of the present application are executed.

[0214] It should be noted that the computer-readable storage medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable computer program instructions. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The computer program instructions contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0215] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by a device, the device can realize functions including: obtaining a training material texture image and an actual surface morphology map corresponding to the training material texture image; processing the training material texture image through the generator in the first network model to generate a predicted surface morphology map corresponding to the training material texture image; comparing the predicted surface morphology map with the actual surface morphology map through the judgement in the first network model to determine the probability that the predicted surface morphology map is the actual surface morphology map corresponding to the training material texture image; training the judgement and generator of the first network model through probability, so that the predicted surface morphology map generated by the trained generator for the training material texture image is the true surface morphology map of the training material texture image.

[0216] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer program instructions, the computer program instructions being stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0217] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution of the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of computer program instructions for enabling an electronic device (which can be a server or a terminal device, etc.) to execute the method according to the embodiment of the present application.

[0218] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application applied for herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not applied for in the present application. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0219] It should be understood that the present application is not limited to the detailed structures, drawings or implementations shown herein, but rather, the present application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a training material texture image of a training material and an actual surface topography image corresponding to the training material texture image; Processing the training material texture image through a generator in the first network model to generate a predicted surface morphology image corresponding to the training material texture image; The predicted surface topography image is compared with the actual surface topography image by a judge in the first network model to determine the probability that the predicted surface topography image is the actual surface topography image corresponding to the training material texture image; The judger and the generator of the first network model are trained by the probability so that the predicted surface morphology map generated by the generator corresponds to the actual surface morphology map of the training material texture image.

2. The method according to claim 1, characterized in that: The method further comprises: Acquire a target mapping relationship between a surface topography image and a tactile stimulation signal; Determining the tactile stimulation signal intensity corresponding to the surface topography image according to the target mapping relationship; The tactile feedback of the tactile stimulation signal intensity is used to simulate the tactile sensation of the material texture.

3. The method according to claim 2, characterized in that: Obtaining a target mapping relationship between a surface topography image and a tactile stimulation signal, including: Determining a first texture gradient curve corresponding to the surface topography; Acquire a first mapping relationship between a texture gradient curve and a tactile perception intensity; A second mapping relationship between the tactile perception intensity and the tactile stimulation signal is acquired so as to determine a target mapping relationship between the surface topography and the tactile stimulation signal according to the first texture gradient curve, the first mapping relationship and the second mapping relationship.

4. The method according to claim 3, characterized in that: Determining the tactile stimulation signal intensity corresponding to the surface topography image according to the target mapping relationship includes: determining a first perception intensity corresponding to the first texture gradient curve through the first mapping relationship; The first tactile stimulation signal intensity corresponding to the first perception intensity is determined through the second mapping relationship, and the first tactile stimulation signal intensity is used as the tactile stimulation signal intensity corresponding to the surface topography image.

5. The method according to claim 3, characterized in that: Acquiring a second mapping relationship between the tactile perception intensity and the tactile stimulation signal includes: Conducting an absolute threshold detection experiment of the tactile stimulation signal on the experimental subject to determine the minimum stimulation signal intensity that causes the tactile sensation of the experimental subject; Conducting a difference threshold experiment of the tactile perception signal on the experimental subject to determine the minimum stimulation signal intensity interval at which the experimental subject feels a change in stimulation intensity after induced tactile sensation; Performing a tactile stimulation amplitude experiment on the experimental subject with the minimum stimulation signal intensity as a starting value and the minimum stimulation signal intensity interval as an intensity interval, so as to determine the tactile perception intensity corresponding to each experimental subject under different stimulation intensities; The tactile perception intensity corresponding to each experimental object under different stimulation intensities is fitted to determine a second mapping relationship between the tactile perception intensity and the tactile stimulation signal.

6. The method according to claim 3, characterized in that: Acquiring a first mapping relationship between a texture gradient curve and the tactile perception intensity includes: Determine an actual texture gradient curve of an actual surface topography image of the training material texture image; Obtaining the actual tactile perception intensity corresponding to the training material texture image; Processing the actual texture gradient curve of the training material texture image through a second network model to predict the predicted tactile perception intensity corresponding to the training material texture image; The second network model is trained using the actual tactile perception intensity and the predicted tactile perception intensity to obtain a trained second network model, wherein the trained second network model is used to describe the first mapping relationship.

7. A tactile rendering method, characterized in that: include: Acquire a second material texture image of a second material; Determine a second surface topography image corresponding to the second material texture image; Acquire a target mapping relationship between a surface topography image and a tactile stimulation signal; Determining, according to the target mapping relationship, a second tactile stimulation signal intensity corresponding to the second surface topography; The second tactile stimulation signal intensity is used to provide tactile feedback to the second object, so as to simulate the tactile sensation when the second object touches the second material.

8. The method according to claim 7, characterized in that: Determining a second surface topography image corresponding to the second material texture image includes: Acquire a training material texture image and an actual surface topography image corresponding to the training material texture image; Processing the training material texture image through a generator in the first network model to generate a predicted surface morphology image corresponding to the training material texture image; The predicted surface topography image is compared with the actual surface topography image by a judge in the first network model to determine the probability that the predicted surface topography image is the actual surface topography image corresponding to the training material texture image; Training the judger and the generator of the first network model by using the probability; The second material texture image is processed by the trained generator to generate the second surface morphology map.

9. An image processing device, characterized in that: include: A training material texture image acquisition module is used to acquire a training material texture image and an actual surface topography image corresponding to the training material texture image; A predicted surface topography map generating module, used for processing the training material texture image through the generator in the first network model to generate a predicted surface topography map corresponding to the training material texture image; A comparison module, configured to compare the predicted surface topography image with the actual surface topography image through a judge in the first network model to determine a probability that the predicted surface topography image is an actual surface topography image corresponding to the training material texture image; A training module is used to train the judger and the generator of the first network model through the probability so that the predicted surface morphology map generated by the generator corresponds to the actual surface morphology map of the training material texture image.

10. A tactile rendering device, characterized in that: include: A second material texture image acquisition module, used to acquire a second material texture image of a second material; A second surface topography map, used to determine a second surface topography map corresponding to the second material texture image; A target mapping relationship acquisition module is used to acquire a target mapping relationship between a surface topography image and a tactile stimulation signal; a second tactile stimulation signal strength determination module, configured to determine a second tactile stimulation signal strength corresponding to the second surface topography image according to the target mapping relationship; The second tactile feedback module is used to provide tactile feedback to the second object through the intensity of the second tactile stimulation signal to simulate the tactile sensation when the second object touches the second material.

11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the image processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

Images