Image display method and device and electronic equipment

Abstract

The invention provides an image display method and apparatus, and an electronic device, applied to an augmented reality wearable device, and determining a target special effect in response to a special effect adding instruction for a current scene; generating image display data based on the target special effect; and controlling the image display device to generate a special effect display layer, and performing image display on the special effect display layer based on the image display data. According to the mode, the user can check the fusion effect of the current scene and the special effect in real time through the augmented reality wearable device, and the use experience of the user on the augmented reality wearable device is improved.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to an image display method, device and electronic equipment. Background Art

[0002] In augmented reality (AR) technology, AR glasses, as an emerging smart wearable device, can provide users with a rich interactive experience by superimposing virtual information and real-world scenes. AR glasses are usually equipped with a camera, and users can take photos or videos through the camera at any time when wearing AR glasses. In related technologies, users usually use image processing software to process the photos or videos taken with AR glasses based on their needs. In this method, the user's operation is relatively complicated, which makes the user's experience of using AR glasses poor. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an image display method, device and electronic device, so that users can perform image processing on images or videos taken by augmented reality wearable devices in real time, and can view the fusion effect of the current scene and special effects in real time through the image reality device, thereby improving the user's experience of using augmented reality wearable devices.

[0004] In a first aspect, an embodiment of the present invention provides an image display method, which is applied to a controller of an augmented reality wearable device; the augmented reality wearable device also includes an image display device connected to the controller; the method includes: determining a target special effect in response to a special effect adding instruction for a current scene; generating image display data based on the target special effect; controlling the image display device to generate a special effect display layer, and displaying an image on the special effect display layer based on the image display data.

[0005] Furthermore, the above-mentioned augmented reality wearable device also includes an audio acquisition device; in response to a special effect adding instruction for the current scene, the step of determining the target special effect includes: in response to the audio acquisition device receiving audio data, performing speech recognition processing on the audio data; based on the recognition result, determining the target special effect.

[0006] Furthermore, the above-mentioned step of controlling the image display device to generate a special effect display layer and displaying an image on the special effect display layer based on the image display data includes: controlling the image display device to generate a new layer in the waveguide display area formed by the image display device, rendering the new layer based on the image display data, and displaying the rendered image.

[0007] Furthermore, the above-mentioned augmented reality wearable device also includes an image acquisition device connected to the controller; the step of generating image display data based on the target special effect includes: controlling the image acquisition device to shoot the current scene to generate an initial image; performing image processing on the initial image based on the image processing method corresponding to the target special effect to generate a target image; and determining the image display data based on the target image and the initial image.

[0008] Furthermore, the above method also includes: determining multiple image areas in the initial image, and determining the target area from the multiple image areas; performing image processing on the initial image based on the image processing method corresponding to the target special effect, and the step of generating the target image includes: performing image processing on the initial image based on the image processing method corresponding to the target special effect and the target area.

[0009] Furthermore, the above step of determining multiple image areas in the initial image includes: converting the initial image into a grayscale image, performing contour recognition processing on the grayscale image to obtain at least one contour; and determining multiple image areas in the initial image based on the at least one contour.

[0010] Furthermore, the above step of determining the target area from multiple image areas includes: responding to an area selection instruction, determining the target area from multiple image areas based on the instruction content of the area selection instruction; the area selection instruction is generated based on voice data or gesture data.

[0011] Furthermore, the above-mentioned step of performing image processing on the initial image based on the image processing method corresponding to the target special effect and the target area includes: determining the area to be processed in the initial image based on the target special effect and the target area; the area to be processed includes the target area and\or the adjacent area of ​​the target area; performing image processing on the area to be processed based on the image processing method corresponding to the target special effect.

[0012] Furthermore, the above-mentioned controller includes a main control unit and a neural network processing unit; the step of performing image processing on the initial image based on the image processing method corresponding to the target special effect to generate the target image includes: sending the initial image to the neural network processing unit through the main control unit; inputting the initial image into a pre-trained image processing model through the neural network processing unit, processing the initial image based on the image processing method corresponding to the target special effect through the image processing model, and outputting the target image.

[0013] Furthermore, the step of determining the image display data based on the target image and the initial image includes: determining the difference between the pixel value of the first pixel and the pixel value of the second pixel for a first pixel in the target image and a second pixel corresponding to the first pixel in the initial image; and generating the image display data based on the difference.

[0014] Furthermore, the above method also includes: updating the target special effect in response to a special effect update instruction; updating the image display data based on the updated target special effect; and controlling the image display device to display the image on the special effect display layer based on the updated image display data.

[0015] Furthermore, after the image display device is controlled to generate a special effect display layer and an image is displayed on the special effect display layer based on the image display data, the method further includes: saving the target image in response to an image confirmation instruction.

[0016] Furthermore, the step of controlling the image acquisition device to shoot the current scene and generate an initial image includes: in response to a video shooting instruction, controlling the image acquisition device to shoot the current scene at a preset frequency to generate a video frame sequence; determining the first video frame in the video frame sequence as the initial image; the method also includes: in the process of generating the video frame sequence, in response to generating a video frame, performing image processing on the video frame based on an image processing method corresponding to a target special effect to generate a target image corresponding to the video frame; determining current image display data based on the target image corresponding to the video frame and the video frame, and controlling the image display device to display the image based on the current image display data.

[0017] Furthermore, the above-mentioned target special effects include at least one of the following: filter special effects, virtual props special effects and beauty special effects.

[0018] In a second aspect, an embodiment of the present invention provides an image display device, which is arranged in a controller of an augmented reality wearable device; the augmented reality wearable device also includes an image display device connected to the controller; the device includes: a target special effect determination module, which is used to determine the target special effect in response to a special effect adding instruction for the current scene; an image display data generation module, which is used to generate image display data based on the target special effect; and an image display module, which is used to control the image display device to generate a special effect display layer, and display an image on the special effect display layer based on the image display data.

[0019] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned image display method.

[0020] In a fourth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image display method.

[0021] The embodiments of the present invention bring the following beneficial effects:

[0022] The above-mentioned image display method, device and electronic device are applied to augmented reality wearable devices, and in response to the special effect adding instruction for the current scene, the target special effect is determined; based on the target special effect, image display data is generated; the image display device is controlled to generate a special effect display layer, and the image is displayed on the special effect display layer based on the image display data. This method allows users to view the fusion effect of the current scene and the special effect in real time through the augmented reality wearable device, thereby improving the user experience of the augmented reality wearable device.

[0023] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0024] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A flowchart of an image display method provided by an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a mark representing an image area provided by an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of an initial image, a target image, and an image displayed by an image display device provided in an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of the system architecture of AR glasses provided by an embodiment of the present invention;

[0030] Figure 5 A flowchart of another image display method provided by an embodiment of the present invention;

[0031] Figure 6 A schematic structural diagram of an image display device provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0033] Figure Number:

[0034] 21-ID 1; 22-ID 2. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0036] In augmented reality (AR) technology, AR glasses, as an emerging smart wearable device, provide users with a rich interactive experience by superimposing virtual information and real-world scenes. However, AR glasses have some limitations in image processing, such as real-time high-quality photo editing functions, which still face challenges. With the development of artificial intelligence (AI) technology, large models have been widely used in the field of image processing. Photo editing applications supported by large models usually require a lot of background computing and high-performance equipment support, which limits their application scenarios and practicality on mobile AR glasses.

[0037] Based on this, an embodiment of the present invention provides an image display method, device and electronic device, which can be applied to image or video shooting scenarios of devices based on augmented reality technology.

[0038] In a possible implementation, an embodiment of the present invention provides an image display method. The method is applied to a controller of an augmented reality wearable device. The augmented reality wearable device also includes an image display device connected to the controller. The above-mentioned augmented reality wearable device may be AR glasses, AR helmets, etc., which are not limited here. The above-mentioned image display device generally includes a projection part and a display part, and the projection part projects the virtual image onto the display part, and the user can see the virtual image on the display part. The above-mentioned display part generally allows the user to see the real scene through the display part. The picture actually seen by the user is a superposition of the real scene and the virtual image. The above-mentioned projection part may be a projector (also called "optical machine"), and the display part may be implemented by optical elements, which may be specifically set according to the needs and are not limited here.

[0039] like Figure 1 As shown, the method comprises the following steps:

[0040] Step S102, in response to a special effect adding instruction for the current scene, determining a target special effect.

[0041] The above-mentioned special effect adding instruction can be generated by the user through voice or gesture. For example, the user can usually generate a special effect adding instruction through voice, for example, the user can say "add a filter to the photo", etc. The target special effect can be determined by performing language recognition on the voice data generated by the user. It is also possible to display multiple special effect corresponding logos through the image display device after the user indicates the need to take a photo or video, such as triggering the generation of a data collection instruction, so that the user can select the desired target special effect through gestures.

[0042] The above-mentioned target special effects can usually be one or more, and specifically can be adding filter special effects, adding virtual props special effects, adding beauty special effects, etc.

[0043] Among them, in the present application, adding virtual props special effects can be: adding accessories / pendants to the display interface, such as hats, glasses, scarves, animal ears, funny beards, etc.; it can also be adding ornaments to the display interface, such as vases, ornaments, paintings, etc., or it can be adding element special effects to the display interface, such as flames, smoke, bubbles, petals, rain, snow, lightning, etc.

[0044] Add filter effects such as retro filter, film filter, fresh filter, European and American filter, black and white filter, warm filter, cold filter, movie filter, transparent filter, food filter, film filter, etc.

[0045] By adding beauty effects, you can: perform some operations on the facial area such as face slimming, body slimming, eye enlargement, nose slimming, freckle removal, whitening, local brightening, color change, and blemish removal.

[0046] Step S104: Generate image display data based on the target special effect.

[0047] According to the determined target special effect, a target special effect image is generated, and the target special effect image is fused with the current scene to generate image display data.

[0048] Specifically, in one embodiment of the present application, in order to realize adding virtual props special effects, target special effects can be added to people or objects in the real scene based on the real scene. Since the user can see the real scene through the image display device of the wearable device, it is only necessary to add virtual props special effects on the display interface through the imaging device, and the user's vision will display a picture in which the virtual props and the display scene are integrated.

[0049] Specifically, taking AR glasses as an example, in response to an instruction to add special effects for the current scene, the processor issues an instruction to control the optical machine to generate a virtual prop pattern. The optical machine projects the virtual pattern onto the waveguide and can adjust the position and size of the virtual prop pattern as needed, so that the human eye can see the effect of superimposed display of the display scene and the virtual prop pattern through the waveguide lens.

[0050] In another embodiment of the present application, in order to realize the filter special effect, it is necessary to obtain the image of the current scene, and uniformly adjust the pixels of the entire image of the current scene. Specifically, by changing the numerical value and ratio of the three channels R, G, and B of each pixel in the image, a preset filter special effect image is generated, and the obtained preset filter special effect image is superimposed with the original current scene graphic, so as to obtain a fused display image with filter special effects. Of course, in the process of superimposed display, the transparency of the filter special effect image in the superimposed process can be adjusted automatically or manually as needed, and the number of superpositions is at least once.

[0051] In another embodiment of the present application, in order to achieve beauty effects, it is necessary to perform graphic processing on the basis of identifying the target object in the current scene. Specifically, the target effect is to perform "beautification" processing on a certain character in the current scene, such as weight loss, big eyes, thin nose, freckle removal, whitening, local brightening, color change, blemish removal, etc. The image processing method corresponding to this target effect needs to determine the display position of the character in the current scene. At this time, it is necessary to capture the display image corresponding to the current scene through an image acquisition device. The image acquisition device in the augmented reality wearable device is usually a camera, and it can also be a CCD sensor and a CMOS image sensor. The image acquisition device is usually able to capture the scene seen by the user wearing the wearable device.

[0052] After the image acquisition device acquires the display image corresponding to the current scene, it is necessary to perform image feature recognition on the initial image to identify the portrait in the display image, as well as the face, facial features, limbs, etc. in the portrait. Then, the image change data corresponding to these areas are determined using the image processing method corresponding to the target special effect. For example, when the target special effect is to whiten only the face area, it is necessary to change the pixel values ​​of the pixels in the area where the face is located in the initial image. Based on the change value of the pixel value of the pixels in this part of the area, the image display data can be generated.

[0053] Step S105 , controlling the image display device to generate a special effect display layer, and performing image display on the special effect display layer based on the image display data.

[0054] The characteristic of augmented reality wearable devices is that there is no need to take pictures. The images of surrounding objects or people can be seen directly by people through transparent lenses. Therefore, augmented reality wearable devices only need to generate special effects picture layers through optical machines, display the special effects to the human eye through waveguides, and superimpose them with the real world seen by the human eye.

[0055] After the image display data is determined, it is necessary to control the image display device to generate a special effect display layer based on the image display data, and to perform image display based on the image display data.

[0056] The user can generate adjustment instructions for the target special effects through voice or gestures. For example, a voice command of "make the brightness higher" can be issued, or a virtual prop displayed by the image display device can be selected through gestures, and then a position in the image displayed by the image device can be selected, indicating that the virtual prop needs to be moved to the selected position.

[0057] In this case, the target image needs to be updated based on the target special effect corresponding to the adjustment instruction. After the target image is updated, the image display data needs to be further updated so that the display effect after the image displayed by the image display device is superimposed with the actual scene can meet the adjustment instruction generated by the user.

[0058] The above-mentioned image display method is applied to an augmented reality wearable device, responding to a special effect adding instruction for the current scene, determining a target special effect; generating image display data based on the target special effect; controlling an image display device to generate a special effect display layer, and displaying an image on the special effect display layer based on the image display data. This method allows users to view the fusion effect of the current scene and the special effect in real time through the augmented reality wearable device, thereby improving the user experience of the augmented reality wearable device.

[0059] The following embodiment provides a specific method for determining a target special effect in response to a special effect adding instruction for a current scene.

[0060] Typically, the augmented reality wearable device also includes an audio acquisition device. The audio acquisition device can collect the voice generated by the user to obtain audio data. In response to the audio acquisition device receiving the audio data, the audio data is subjected to voice recognition processing; then, based on the recognition result, the target special effect is determined. A trigger word can be set in advance, and when the user says this trigger word, the subsequent voice content expresses the target special effect required. For example, the trigger word can be set to "photo editing". If "photo editing" is included in the recognition result, the target special effect can be determined based on the recognition result after the photo editing. If there is no trigger word in the recognition result, no processing is performed.

[0061] The following embodiment provides a specific method for generating image display data based on a target special effect.

[0062] In a specific embodiment, when the target special effect needs to add a virtual prop special effect to the current scene, in response to the special effect adding instruction for the current scene, the processor issues an instruction to control the imaging device to generate a virtual prop pattern. The imaging device projects the virtual pattern onto the display device and can adjust the position and size of the virtual prop pattern as needed, so that the human eye can see the effect of superimposing or fusion display of the display scene and the virtual prop pattern on the display device.

[0063] In a specific embodiment, when the target special effect needs to add a filter special effect to the current scene, the original image captured by the image acquisition device can be uniformly processed by software, and the image effect can be generated directly on the captured original image, or the original image can be copied, and the image effect can be added to the copied image through software, and the image with the added image effect can be superimposed or fused with the original image to generate a target image for display on the display interface.

[0064] In a specific embodiment, when the target special effect needs to add beauty effects to a part of the current scene, such as facial whitening, face slimming, leg slimming, etc., a grayscale image of the current scene can be obtained through an image acquisition device, and the target feature recognition of the area to be processed is performed based on the grayscale image to determine the portrait area to be processed, that is, the contour of the target object is recognized through the acquired grayscale image, and then the original color image and the grayscale image face area after face slimming processing are superimposed or fused.

[0065] Specifically, taking the face slimming and beautifying function as an example, the specific steps are as follows:

[0066] S201, converting the color image to a grayscale image, converting the captured color image to a grayscale image. There are many common grayscale conversion formulas, such as the weighted average method:

[0067] For each pixel in the color image, its RGB values ​​are R, G, and B respectively. The gray value Gray corresponding to the pixel is calculated by the formula Gray = 0.299*R+0.587*G+0.114*B, thereby converting the entire color image into a grayscale image. The purpose of this is to reduce the amount of data and reduce the computational complexity when performing face contour recognition.

[0068] S202, perform face contour recognition, and use a face contour recognition algorithm to process the converted grayscale image. A cascade classifier based on Haar features, a face detection algorithm based on deep learning, etc. can be used; the algorithm detects the contour position of the face on the grayscale image and determines the area where the face is located, such as obtaining the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the rectangular area of ​​the face.

[0069] S203, performing face thinning processing on the identified face contour area of ​​the grayscale image. Some scaling methods are used to achieve an approximate face thinning effect, specifically:

[0070] According to the preset face-thinning ratio, the pixels in the face contour area are scaled. For example, if the preset face-thinning ratio is 0.9 (indicating 10% thinning), then for each pixel (x, y) in the face contour area, its new coordinates (x', y') can be calculated by the formula x'=x*0.9, y'=y*0.9 (the face-thinning ratio can be preset with a fixed value, or it can be adaptively adjusted according to the specific shape and features of the face according to a preset algorithm). In this way, the face is given a thin face visual effect on the image.

[0071] S204, performing image blending on the original color image and the face region of the grayscale image after face thinning processing; specifically, performing a superposition operation on the face region in the original color image (determined according to the coordinate range obtained by the previous face contour recognition) and the face region in the grayscale image after the face thinning processing according to the principle of transparency blending;

[0072] In the process of superposition, a transparency value α (ranging from 0 to 1) is set, for example, α = 0.5; for each pixel point (x, y) in the face area: the RGB value of the pixel point in the original color image is recorded as (R1, G1, B1); the grayscale value of the pixel point at the corresponding position in the face area of ​​the processed grayscale image is recorded as Gray2 (convert it to RGB value, for example, a pseudo RGB value is formed in the form of Gray2, Gray2, Gray2, assuming that the converted RGB value is (R2, G2, B2)); after transparency blending, the new RGB value (R', G', B') of the pixel point can be calculated by the following formula:

[0073] R'=α*R1+(1-α)*R2

[0074] G'=α*G1+(1-α)*G2

[0075] B'=α*B1+(1-α)*B2

[0076] By performing transparency blending operations on each pixel in the face area, the features of the face area of ​​the grayscale image after face slimming processing can be superimposed on the face area of ​​the original color image, thereby achieving a face slimming effect on the color image face.

[0077] It should be noted that the parameters in the above steps (such as face-slimming ratio, transparency value, etc.) can be preset fixed parameters, or can be preset different parameter values ​​corresponding to different levels of face-slimming degree and face-slimming effect.

[0078] The solution of the present application requires less byte space for each pixel of the grayscale image, and in the process of contour recognition, only a simple calculation needs to be performed on each pixel according to the grayscale conversion formula, and there is no need to simultaneously process the complex data relationship of the three RGB channels of the color image. Therefore, the memory and computing power required for contour recognition of the grayscale image are also smaller; when performing transparency blending operations, the pixels of the two areas (the original color image face area and the processed grayscale image face area) are mainly processed accordingly according to the set transparency values, and there is no need to open up a large amount of temporary storage space to store intermediate calculation results like some complex algorithms. Therefore, memory space, computing power and energy consumption can be saved, and the battery life of the device can be effectively improved, overheating can be avoided, and a better user experience can be provided to users.

[0079] In actual applications, users can control the image acquisition device to shoot the current scene through data acquisition instructions. The above data acquisition instructions can be triggered by users through voice or gestures. For example, when the user says "take a photo" or "record a video", the data acquisition instruction is triggered; or when the user makes a gesture with his hand, such as the common "scissor hand" gesture, the data acquisition instruction is triggered. It is also possible that the above voice and gesture are generated by the user at the same time or before and after, which will trigger the generation of data acquisition instructions. It can be set specifically according to needs and is not limited here.

[0080] In a specific embodiment, the data acquisition instruction may be an image capture instruction, which is only used to control the generation of one image, or may be used to control the generation of multiple images, such as a continuous shooting mode used in taking photos.

[0081] In another specific embodiment, the data acquisition instruction may be a video shooting instruction, which is used to control the image acquisition device to shoot a video. Shooting a video is actually a process of continuously shooting multiple images. After the data acquisition instruction is generated, the image acquisition device needs to shoot the current scene to generate an initial image. The scene displayed in the initial image is usually consistent with the scene seen by the user using the wearable device.

[0082] When the data acquisition instruction is a video shooting instruction, in response to the video shooting instruction, the image acquisition device is controlled to shoot the current scene at a preset frequency, generate a video frame sequence, and determine the first video frame in the video frame sequence as the initial image. In the process of generating a video frame sequence, in response to generating a video frame, the video frame is image processed based on the image processing method corresponding to the target special effect to generate a target image corresponding to the video frame, and then based on the target image corresponding to the video frame and the video frame, the current image display data is determined, and the image display device is controlled to display the image based on the current image display data. In this way, frame-by-frame image processing of the video is realized, so that the user can perform image processing on the video frame in real time during the process of shooting the video, thereby improving the user experience.

[0083] In a specific embodiment, when image processing is required for an object in an initial image, multiple image regions in the initial image are first determined, and then a target region is determined from the multiple image regions. Next, image processing is performed on the initial image based on the image processing method corresponding to the target special effect and the target region.

[0084] As described above, a grayscale image can be directly taken as the initial image, or a color image can be taken as the initial image. If the initial image is a grayscale image, contour recognition processing can be directly performed based on the grayscale image to determine the contour in the grayscale image. If the initial image is a color image, the initial image can be converted into a grayscale image, and then the grayscale image can be subjected to contour recognition processing to obtain at least one contour. When performing contour recognition, it is usually determined whether two pixels are at the intersection of contours based on the difference in pixel values ​​between adjacent pixels. When the pixel value difference is greater than a specified value, the position between the two pixels is usually determined as the contour portion. When the specified value is set differently, the number and position of contours obtained after contour recognition processing are usually different. Based on at least one contour, multiple image areas in the initial image are determined. In the initial image, the area surrounded by the contour and the area outside the contour are usually determined as different image areas.

[0085] After determining multiple image areas, the user can select one or more image areas as target areas based on needs. For example, if multiple portraits are displayed in the initial image, the display area of ​​one or several portraits can be selected as the target area. The user can generate an area selection instruction through voice or gesture, and in response to the area selection instruction, the target area is determined from multiple image areas based on the instruction content of the area selection instruction. This method allows users to freely select the area that needs to be processed, thereby improving the user experience.

[0086] In order to let the user know which image regions the image region includes, the identification display data can be determined based on the positions of the multiple image regions in the initial image, and the image display device can be controlled to display the identification based on the identification display data. The identification can indicate the outline of the image region, or indicate the position of the image region, or both. Figure 2 As shown, the rectangular mark corresponding to reference numeral 21 represents mark 1, which is displayed in the image area corresponding to portrait 1; the rectangular mark corresponding to reference numeral 22 represents mark 2, which is displayed in the image area corresponding to portrait 2. The user can select mark 1 or mark 2 through voice or gesture to make the image area corresponding to the mark 1 or mark 2 the target area.

[0087] When the user selects one or more image areas as the target area, it is necessary to determine the area to be processed in the initial image based on the image processing method corresponding to the target special effect and the target area. The area to be processed may be the target area, and may also include the adjacent area of ​​the target area, or only include the adjacent area of ​​the target area.

[0088] The above-mentioned initial image usually displays one or more objects, that is, it has a display area corresponding to each object. There are many types of target special effects, which can be denoising, filter addition, color parameter adjustment, special effect addition, and overall or partial shape adjustment of the object, etc. Different target special effects will result in different determined areas to be processed. For example, when the target special effect is to add a virtual prop along the outline of the target area, such as adding a "headband", the area to be processed is the adjacent area to the target area; when the target special effect is "slimming", since the outline of the target area needs to change, the outline of the adjacent area of ​​the target area also needs to change. At this time, the target area and its adjacent areas are both areas to be processed. After determining the area to be processed, the image processing method corresponding to the target special effect can be used to process the area to be processed.

[0089] In a specific embodiment, the controller may include a main control unit and a neural network processing unit. Among them, the neural network processing unit (NPU) can be implemented by an independent chip. The chip can simulate human neurons and synapses in the circuit. Perform artificial intelligence operations to produce a new type that can respond in a similar way to human intelligence. The initial image and the target special effects can be sent to the neural network processing unit through the main control unit, and then the initial image can be input into a pre-trained image processing model through the neural network processing unit. The image processing model processes the initial image based on the image processing method corresponding to the target special effects, and outputs the target image. The above-mentioned image processing model is usually established based on a neural network and is obtained through a large amount of data training, and has high image processing quality and speed.

[0090] In a specific embodiment, the determination of image display data based on the target image and the initial image can be achieved in the following manner: first, the difference between the target image and the initial image is determined, and the difference between the pixel value of the first pixel and the pixel value of the second pixel can be determined for the first pixel in the target image and the second pixel corresponding to the first pixel in the initial image. The position of the first pixel in the target image and the position of the second pixel in the initial image are usually the same, such as the first pixel is the pixel in the first row and the fifth column in the target image, and the second pixel corresponding to the first pixel is the pixel in the first row and the fifth column in the initial image. Then, based on the difference, the image display data is generated. The position of each pixel in the target image and the difference corresponding to the pixel can be counted, and this information can be stored as image display data for subsequent image display on the image display device.

[0091] The following embodiments provide a specific method of controlling an image display device to generate a special effect display layer and performing image display on the special effect display layer based on image display data.

[0092] When controlling the image display device to display an image, the image display device is usually controlled to generate a new layer in the waveguide display area formed by the image display device, and the new layer is rendered based on the image display data to display the rendered image.

[0093] The waveguide display area mainly includes a coupling-in area and a coupling-out area. The coupling-in area is used to couple the light beam of the micro-projection optical machine into the waveguide plate, so that the light beam satisfies the condition of total reflection propagation in the waveguide plate. The coupling-out area is used to couple the light beam of total reflection propagation out of the waveguide plate and transmit it to the human eye. The coupling-in area can be a reflector, a prism, a relief grating, a volume holographic grating, etc., while the coupling-out area can be an array of semi-transparent and semi-reflective mirrors, a relief grating, a volume holographic grating, etc.1. The micro-projection optical machine will generate a corresponding light beam based on the image display data, and transmit it to the human eye through the coupling effect of the coupling-in area and the coupling-out area, so that the human eye can see the corresponding image.

[0094] like Figure 3 As shown in the figure, the left side is the initial image, the middle is the target image, and the target image adds a hat to the portrait shown in the initial image. The right side is the image generated based on the image display data. The user will see a hat displayed on the head of the person in the real scene.

[0095] In actual applications, users can also generate small update instructions through voice or gestures. In response to the special effects update instruction, the target special effects need to be updated. And based on the updated target special effects, the image display data is updated. When the updated target special effects need to be generated based on the display content of the current scene, the update process of the image display data is usually also implemented by the neural network processing unit, which can quickly respond to the user's image processing needs. Furthermore, it is necessary to control the image display device to display the image on the special effects display layer based on the updated image display data, so that the user can see the display effect of the updated target image.

[0096] The user can also generate an image confirmation command through voice or gesture, and save the target image in response to the image confirmation command. Specifically, the target image can be saved to a storage control preset in the controller, or the target image can be transferred to a cloud storage space, which can meet the user's various storage needs.

[0097] The embodiment of the present invention also provides another image display method. Figure 1 This method takes AR glasses in augmented reality wearable devices as an example, and uses AI big models to enable AR smart glasses to provide users with real-time high-quality image editing and processing (also known as "photo editing") without the need for post-processing, which significantly improves the efficiency and quality of image editing and processing.

[0098] This method requires the camera in the AR smart glasses (equivalent to the above-mentioned "image acquisition device") to identify the scene in reality, so as to obtain object information (such as people or objects), and then the user can determine the image modification or fusion that needs to be made according to personal needs. Specifically, image processing can be used for beautification or adding special effects, such as adding a virtual hat to the character image, generating anthropomorphic animation of the character image, etc. Then, based on the identified information and the required image modification or fusion, the corresponding effect is generated, and the user can view the effect in real time through the image display device. The implementation method is usually to form a new layer in the waveguide display area, and visually superimpose it with the object image in the real world, so as to form the visual effect required by the user.

[0099] The method is applied to AR smart glasses equipped with sensors and cameras, which integrate an AI module (also known as an "AI acceleration chip") that uses a large model to process image data and a real-time visualization engine. The AI ​​module generates dynamic photo editing effects and provides detailed image editing tools and filters (i.e., special effects layers). These photo editing effects and tools are displayed in real time on the AR smart glasses, improving the user experience.

[0100] like Figure 4As shown, AR smart glasses are equipped with high-definition cameras, display screens (also called "lens displays"), processors (i.e. Figure 7 The main control unit includes a micro-projection optical engine, a wireless communication module, and an AI chip connected to the main control unit to accelerate the reasoning process of the AI ​​model and ensure real-time processing capabilities. A storage unit connected to the main control unit is also provided to store the AI ​​model, photo editing applications, and processed image data.

[0101] like Figure 5 As shown, the method is implemented by the following steps:

[0102] 1. Image capture: The high-definition camera of the AR smart glasses is used to capture (also called "capture") images in the user's field of view in real time. The captured image is transmitted to the main control unit, which converts the image information into two-dimensional information of the object, that is, identifies the outline of the object. Specifically, the outline of the object can be identified by converting the image into grayscale data, and based on the identified outline, the spatial two-dimensional coordinate information of each object in the image is determined, that is, where the object is located in the image.

[0103] This method does not need to obtain information such as the color and light of the object, so the amount of data obtained is very small. Compared with real-time video or image processing, the amount of data is much smaller. Therefore, the requirements for the device processor, storage space, and heat dissipation will be reduced, and the battery life of the device will also be improved.

[0104] 2. AI photo editing: An image processing module that integrates a pre-trained AI photo editing model to perform image enhancement, filter application, blemish repair, and other processing.

[0105] Specifically, the main control unit processor calls the AI ​​chip (equivalent to the above-mentioned "image processing module", which can be realized by the above-mentioned "neural network processing unit") and uses the pre-trained image editing model to process the image, including but not limited to color adjustment, denoising, sharpening, beautification, blemish repair, various visual templates / special effects, etc. Of course, users can also manually select templates according to their needs - such as skin smoothing, face slimming, leg slimming, denoising, one-click beautification, adding special effects, etc.

[0106] In addition to AI photo editing, it can also obtain light information about the environment or objects, such as light intensity information, and can adjust it according to the system default or user-defined brightness range. When the brightness is low, increase the brightness to the preset interval brightness, and when the brightness is high, lower the brightness to the preset interval brightness.

[0107] 4. Result display: The processed image is presented to the user through the display module of the AR glasses for real-time preview. Usually, the captured real image (equivalent to the above-mentioned "initial image") is superimposed with the special effects and displayed to the user in real time.

[0108] 5. User interaction: Users interact through voice commands, gestures, etc. to adjust the photo editing effects.

[0109] 6. Image saving: After user confirmation, the processed image can be saved to the local storage unit or directly uploaded to the cloud storage through the wireless communication module.

[0110] This method can be applied to the following scenarios:

[0111] 1 The user uses an AR smart glasses with an integrated AI photo editing model and starts the image processing function through the voice command "photo editing". The AI ​​model automatically beautifies and enhances the images captured in real time. The processed images are displayed through the glasses' display screen. Users can further adjust the effects and save them through gestures.

[0112] 2 At tourist attractions, users wear AR smart glasses to take photos and videos. The system automatically recognizes the scene and applies appropriate filters and adjustments. The processed high-quality photos or videos can be directly uploaded to the user's social media account to share the travel experience in real time.

[0113] This method uses the integration of AR glasses and AI large models, allowing users to perform instant photo editing on images captured in real time, improving user experience and ease of operation. It overcomes the problem of long-term hand-held operation when taking real-time photos with mobile phones or photographic equipment, and overcomes the problem of jitter and fatigue caused by long-term hand-held operation, as well as poor shooting effects and inconvenience caused by jitter. The implementation of this method on mobile AR glasses expands the application scenarios and market potential of AR glasses.

[0114] This method utilizes the above-mentioned overlay layer method to achieve real-time interaction between users and the external environment, while allowing users to adjust the type and parameters of special effects, view and adjust image effects in real time. Users can perform DIY creations according to their own ideas, which improves the user experience and can even perform secondary creations to form functions similar to movies or short videos. It is suitable for real-time social interaction, social media content creation, education and training and other fields.

[0115] Users can also achieve social interaction based on the above method. For example, they can use the camera to obtain images or videos of the environment or objects, and fuse them with the special effects information set by the user to generate new images or video information, and transmit them to other users for sharing through the device's wireless module - WIFI or Bluetooth or cellular data; or upload them to a server or a bound social media account for publishing.

[0116] Users can also implement education and training based on the above method. For example, after generating new image or video information using the above method, the information is uploaded to the network cloud or local server, and then distributed to other users' AR devices via the wireless network, thereby realizing online one-to-many knowledge sharing.

[0117] For the above method embodiments, see Figure 6 An image display device is shown, which is arranged in a controller of an augmented reality wearable device; the augmented reality wearable device also includes an image display device connected to the controller; the device includes:

[0118] A target special effect determination module 602 is used to determine a target special effect in response to a special effect adding instruction for a current scene;

[0119] An image display data generating module 604 is used to generate image display data based on target special effects;

[0120] The image display module 606 is used to control the image display device to generate a special effect display layer, and to display an image on the special effect display layer based on the image display data.

[0121] The above-mentioned image display device is applied to an augmented reality wearable device, responds to a special effect adding instruction for the current scene, determines a target special effect; generates image display data based on the target special effect; controls the image display device to generate a special effect display layer, and displays the image on the special effect display layer based on the image display data. This method allows users to view the fusion effect of the current scene and the special effect in real time through the augmented reality wearable device, thereby improving the user experience of the augmented reality wearable device.

[0122] The above-mentioned augmented reality wearable device also includes an audio acquisition device; the target special effect determination module is also used to: in response to the audio acquisition device receiving audio data, perform speech recognition processing on the audio data; and determine the target special effect based on the recognition result.

[0123] The above-mentioned image display module is also used to: control the image display device to generate a new layer in the waveguide display area formed by the image display device, render the new layer based on the image display data, and display the rendered image.

[0124] The above-mentioned image display data generation module is also used to: control the image acquisition device to shoot the current scene to generate an initial image; perform image processing on the initial image based on the image processing method corresponding to the target special effect to generate a target image; and determine the image display data based on the target image and the initial image.

[0125] The above-mentioned device also includes: a target area determination module, which is used to determine multiple image areas in the initial image and determine the target area from the multiple image areas; the above-mentioned image display data generation module is also used to: perform image processing on the initial image based on the image processing method corresponding to the target special effect and the target area.

[0126] The target area determination module is also used to: convert the initial image into a grayscale image, perform contour recognition processing on the grayscale image to obtain at least one contour; and determine multiple image areas in the initial image based on the at least one contour.

[0127] The target area determination module is further used to: determine the target area from multiple image areas in response to an area selection instruction based on the instruction content of the area selection instruction; the area selection instruction is generated based on voice data or gesture data.

[0128] The above-mentioned image display data generation module is also used to: determine the area to be processed in the initial image based on the target special effects and the target area; the area to be processed includes the target area and\or the adjacent area of ​​the target area; based on the image processing method corresponding to the target special effects, perform image processing on the area to be processed.

[0129] The above-mentioned controller includes a main control unit and a neural network processing unit; the above-mentioned image display data generation module is also used to: send the initial image to the neural network processing unit through the main control unit; input the initial image into a pre-trained image processing model through the neural network processing unit, process the initial image through the image processing model based on the image processing method corresponding to the target special effect, and output the target image.

[0130] The image display data generation module is also used to: determine the difference between the pixel value of the first pixel and the pixel value of the second pixel corresponding to the first pixel in the target image and the second pixel in the initial image; and generate image display data based on the difference.

[0131] The above-mentioned device also includes: a special effect update module, which is used to update the target special effect in response to the special effect update instruction; an image display data update module, which is used to update the image display data based on the updated target special effect; and an image update display module, which is used to control the image display device to display the image on the special effect display layer based on the updated image display data.

[0132] The above device also includes: a target image saving module, which is used to save the target image in response to the image confirmation instruction.

[0133] The above-mentioned image display data generation module is also used to: respond to a video shooting instruction, control the image acquisition device to shoot the current scene at a preset frequency to generate a video frame sequence; determine the first video frame in the video frame sequence as the initial image; the above-mentioned method also includes: in the process of generating a video frame sequence, in response to generating a video frame, perform image processing on the video frame based on the image processing method corresponding to the target special effect to generate a target image corresponding to the video frame; determine the current image display data based on the target image corresponding to the video frame and the video frame, and control the image display device to display the image based on the current image display data.

[0134] The above-mentioned target special effects include at least one of the following: filter special effects, virtual props special effects and beauty special effects.

[0135] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned image display method.

[0136] See also Figure 7 As shown, the electronic device includes a processor 100 and a memory 101. The memory 101 stores machine executable instructions that can be executed by the processor 100. The processor 100 executes the machine executable instructions to implement the above-mentioned image display method.

[0137] Further, Figure 7 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected via the bus 102 .

[0138] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0139] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 100. The above processor 100 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams of the invention in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method invented in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method of the above embodiment in combination with its hardware.

[0140] This embodiment further provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image display method.

[0141] An image display method, device and electronic device provided by the embodiments of the present invention include a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the methods described in the above method embodiments.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0143] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0144] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0145] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0146] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image display method, characterized in that: The method is applied to a controller of an augmented reality wearable device; the augmented reality wearable device also includes an image display device connected to the controller; the method includes: In response to a special effect adding instruction for the current scene, determining a target special effect; Based on the target special effect, generate image display data; The image display device is controlled to generate a special effect display layer, and an image is displayed on the special effect display layer based on the image display data.

2. The method according to claim 1, characterized in that The augmented reality wearable device also includes an audio acquisition device; In response to a special effect adding instruction for the current scene, the step of determining a target special effect comprises: In response to the audio data being received by the audio acquisition device, performing speech recognition processing on the audio data; Based on the recognition results, the target special effects are determined.

3. The method according to claim 1, characterized in that The step of controlling the image display device to generate a special effect display layer, and displaying an image on the special effect display layer based on the image display data comprises: The image display device is controlled to generate a new layer in a waveguide display area formed by the image display device, and the new layer is rendered based on the image display data to display the rendered image.

4. The method according to claim 1, characterized in that: The augmented reality wearable device also includes an image acquisition device connected to the controller; Based on the target special effect, the step of generating image display data comprises: Controlling the image acquisition device to capture the current scene and generate an initial image; Performing image processing on the initial image based on an image processing method corresponding to the target special effect to generate a target image; Image display data is determined based on the target image and the initial image.

5. The method according to claim 4, characterized in that The method further comprises: Determine a plurality of image regions in the initial image, and determine a target region from the plurality of image regions; The step of performing image processing on the initial image based on the image processing method corresponding to the target special effect to generate the target image comprises: Based on the image processing method corresponding to the target special effect and the target area, the initial image is processed.

6. The method according to claim 5, characterized in that The step of determining a plurality of image regions in the initial image comprises: Converting the initial image into a grayscale image, and performing contour recognition processing on the grayscale image to obtain at least one contour; Based on the at least one contour, a plurality of image regions in the initial image are determined.

7. The method according to claim 5, characterized in that The step of determining a target area from the plurality of image areas comprises: In response to an area selection instruction, a target area is determined from the multiple image areas based on instruction content of the area selection instruction; the area selection instruction is generated based on voice data or gesture data.

8. The method according to claim 5, characterized in that The step of performing image processing on the initial image based on the image processing method corresponding to the target special effect and the target area includes: Based on the target special effect and the target area, determining a region to be processed in the initial image; the region to be processed includes the target area and / or an adjacent area of ​​the target area; Based on the image processing method corresponding to the target special effect, image processing is performed on the area to be processed.

9. The method according to claim 4, characterized in that The controller includes a main control unit and a neural network processing unit; The step of performing image processing on the initial image based on the image processing method corresponding to the target special effect to generate the target image includes: Sending the initial image to the neural network processing unit through the main control unit; The initial image is input into a pre-trained image processing model through the neural network processing unit, and the initial image is processed by the image processing model based on the image processing method corresponding to the target special effect to output the target image.

10. The method according to claim 4, characterized in that The step of determining image display data based on the target image and the initial image comprises: For a first pixel in the target image and a second pixel in the initial image corresponding to the first pixel, determining a difference between a pixel value of the first pixel and a pixel value of the second pixel; Based on the difference, image display data is generated.

11. The method according to claim 1, characterized in that: The method further comprises: In response to the special effect update instruction, updating the target special effect; Based on the updated target special effect, updating the image display data; The image display device is controlled to display an image on the special effect display layer based on the updated image display data.

12. The method according to claim 4, characterized in that The image display device is controlled to generate a special effect display layer, and after the special effect display layer performs image display based on the image display data, the method further includes: In response to the image confirmation instruction, the target image is saved.

13. The method according to claim 4, characterized in that The step of controlling the image acquisition device to capture the current scene and generate an initial image includes: In response to the video shooting instruction, controlling the image acquisition device to shoot the current scene at a preset frequency to generate a video frame sequence; Determine the first video frame in the video frame sequence as an initial image; The method further comprises: In the process of generating a video frame sequence, in response to generating a video frame, performing image processing on the video frame based on the image processing method corresponding to the target special effect to generate a target image corresponding to the video frame; Based on the target image corresponding to the video frame and the video frame, current image display data is determined, and the image display device is controlled to display an image based on the current image display data.

14. The method according to claim 4, characterized in that The target special effects include at least one of the following: filter special effects, virtual prop special effects and beauty special effects.

15. An image display device, characterized in that: The device is arranged on a controller of an augmented reality wearable device; the augmented reality wearable device further comprises an image display device connected to the controller; the device comprises: A target special effect determination module, for determining a target special effect in response to a special effect adding instruction for a current scene; An image display data generating module, used for generating image display data based on the target special effect; The image display module is used to control the image display device to generate a special effect display layer, and to display an image on the special effect display layer based on the image display data.

16. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image display method according to any one of claims 1 to 14.

17. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the image display method according to any one of claims 1 to 14.

Images