Image processing method and device, electronic equipment and readable storage medium

Abstract

Embodiments of the present application provide an image processing method and device, electronic equipment and readable storage medium, and relate to the technical field of image processing. Embodiments of the present application obtain an image frame to be processed and a target displacement map corresponding to the image frame, determine pixel displacement values of second pixel points in the image frame according to displacement offsets stored in color channels of each first pixel point in the target displacement map, then determine sampling pixel points corresponding to each second pixel point in the image frame according to the pixel displacement values of each second pixel point, and then replace pixel information of each second pixel point with pixel information of the sampling pixel points corresponding to each second pixel point, so that a fault special effect with displacement can be realized, visual experience is enriched, and user stickiness is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the development of computer technology, smart terminals have been widely used, allowing users to take photos and videos. To meet users' increasingly personalized demands for photos, adding special effects to photos and videos has become a popular way to enhance images. However, current special effects processing still needs improvement, which is not conducive to increasing user engagement. Summary of the Invention

[0003] Based on the above research, the present invention provides an image processing method, apparatus, electronic device and readable storage medium, which realizes the fault effect of displacement, enriches the visual experience and helps to improve user stickiness.

[0004] Embodiments of the present invention can be implemented in the following ways:

[0005] In a first aspect, embodiments of the present invention provide an image processing method, comprising:

[0006] Obtain the image frame to be processed and the target permutation map corresponding to the image frame; the color channel of each first pixel in the target permutation map stores the displacement offset;

[0007] The pixel displacement value of each second pixel in the image frame is determined based on the displacement offset in the color channel of each first pixel.

[0008] Based on the pixel displacement value of each second pixel, the sampling pixel point corresponding to each second pixel point in the image frame is determined;

[0009] Replace the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel.

[0010] In an optional implementation, obtaining the target permutation map corresponding to the image frame includes:

[0011] Obtain the initial permutation map;

[0012] The aspect ratio of the image frame is obtained based on its resolution.

[0013] Based on the aspect ratio of the image frame, the initial permutation map is cropped to obtain the target permutation map.

[0014] In an optional implementation, obtaining the initial permutation map includes:

[0015] Based on the frame number of the image frame, an initial permutation map is determined in the preset permutation map sequence frame that matches the frame number of the image frame; the preset permutation map sequence frame includes multiple permutation maps, each permutation map has a frame number, and the displacement offset stored in the color channel of the first pixel in each permutation map is different.

[0016] In an optional implementation, determining the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channel of each first pixel includes:

[0017] The offset coefficient of each second pixel in the image frame is obtained based on the preset offset threshold and scaling factor.

[0018] For each second pixel in the image frame, based on the position information of the second pixel, a target first pixel whose position information matches the position information of the second pixel is obtained in the target permutation map;

[0019] The pixel displacement value of the second pixel is obtained based on the displacement offset in the color channel of the first target pixel and the offset coefficient.

[0020] In an optional implementation, after acquiring the image frame to be processed, the method further includes:

[0021] Obtain the color information of each second pixel in each color channel and the color offset of each color channel;

[0022] For each color channel, based on the color offset of the color channel, determine the color offset pixel point in the image frame corresponding to each second pixel point, and replace the color information of each second pixel point in the color channel with the color information of the color offset pixel point corresponding to each second pixel point in the color channel.

[0023] In an optional implementation, after replacing the pixel information of each second pixel with the pixel information of the sampled pixel corresponding to each second pixel, the method further includes:

[0024] For each second pixel in the image frame, if the position information of the second pixel in the first direction is less than a first preset value, then the target pixel corresponding to the second pixel is determined according to the sum of the preset displacement value and the position information, and the pixel information of the second pixel is replaced with the pixel information of the target pixel.

[0025] If the position information of the second pixel in the first direction is greater than the second preset value, then the target pixel corresponding to the second pixel is determined according to the difference between the position information and the displacement value, and the pixel information of the second pixel is replaced with the pixel information of the target pixel, wherein the second preset value is greater than the first preset value.

[0026] In an optional implementation, determining the target pixel corresponding to the second pixel based on the sum of a preset displacement value and the position information includes:

[0027] Based on the frame number of the image frame, a target jitter value whose ordinal number matches the frame number is determined in a preset jitter value sequence; the jitter value sequence includes multiple jitter values, each jitter value having a unique ordinal number;

[0028] The target pixel corresponding to the second pixel is obtained by summing the displacement value, the target jitter value, and the position information.

[0029] The step of determining the target pixel corresponding to the second pixel based on the difference between the position information and the displacement value includes:

[0030] The target pixel corresponding to the second pixel is obtained based on the difference between the position information, the displacement value, and the target jitter value.

[0031] In an optional implementation, after replacing the pixel information of each second pixel with the pixel information of the sampled pixel corresponding to each second pixel, the method further includes:

[0032] For each second pixel in the image frame, the second pixel is sampled in the second direction according to a preset blur radius to obtain a first sampled pixel.

[0033] Based on the blur radius, each of the first sampled pixels is sampled in the second direction to obtain the second sampled pixels;

[0034] The target pixel information of the second pixel is obtained based on the average value of the pixel information of each of the first sampled pixels and each of the second sampled pixels.

[0035] In an optional implementation, after replacing the pixel information of each second pixel with the pixel information of the sampled pixel corresponding to each second pixel, the method further includes:

[0036] Based on the frame number of the image frame, a target luminous layer in a preset luminous foreground sequence frame whose frame number matches that of the image frame is determined; the preset permutation map sequence frame includes multiple luminous layers, each luminous layer having a frame number;

[0037] Add the target luminescence layer to the image frame to obtain the image frame with the target luminescence layer added.

[0038] Add a preset noise layer to the image frame after adding the target glow layer.

[0039] In a second aspect, embodiments of the present invention provide an image processing apparatus, the image processing apparatus comprising:

[0040] The image acquisition module is used to acquire the image frame to be processed and the target permutation map corresponding to the image frame; the color channel of each first pixel in the target permutation map stores the displacement offset.

[0041] The offset determination module is used to determine the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channel of each first pixel.

[0042] The sampling determination module is used to determine the sampling pixel point in the image frame corresponding to each of the second pixel points based on the pixel displacement value of each of the second pixel points;

[0043] The pixel determination module is used to replace the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel.

[0044] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method described in any of the foregoing embodiments.

[0045] Fourthly, embodiments of the present invention provide a readable storage medium, the readable storage medium including a computer program, wherein the computer program, when running, controls the electronic device containing the readable storage medium to execute the image processing method described in any of the foregoing embodiments.

[0046] The image processing method, apparatus, electronic device, and readable storage medium provided in this invention acquire an image frame to be processed and a target permutation map corresponding to the image frame. Based on the displacement offset stored in the color channel of each first pixel in the target permutation map, the pixel displacement value of each second pixel in the image frame is determined. Then, based on the pixel displacement value of each second pixel, the sampling pixel corresponding to each second pixel in the image frame is determined. Finally, the pixel information of each second pixel is replaced with the pixel information of the sampling pixel corresponding to each second pixel. This achieves a fault effect with displacement, enriching the visual experience and improving user engagement. Furthermore, by storing the displacement offset in the color channel of the pixels in the permutation map and obtaining the pixel displacement value of each pixel in the image frame based on the displacement offset, the operation is simple and the response speed is fast, further enhancing user engagement. Attached Figure Description

[0047] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0048] Figure 1 This is a schematic diagram of an electronic device provided in an embodiment of the present invention.

[0049] Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of the present invention.

[0050] Figure 3 This is a permutation diagram provided in an embodiment of the present invention.

[0051] Figure 4 This is a schematic diagram of the cutting of the displacement diagram provided in an embodiment of the present invention.

[0052] Figure 5 This is another permutation diagram provided in an embodiment of the present invention.

[0053] Figure 6 This is another flowchart illustrating the image processing method provided in an embodiment of the present invention.

[0054] Figure 7 This is a displacement fault effect diagram provided in an embodiment of the present invention.

[0055] Figure 8 This is a mixed effect diagram provided for an embodiment of the present invention.

[0056] Figure 9 This is another flowchart illustrating the image processing method provided in an embodiment of the present invention.

[0057] Figure 10 This is a color shift effect diagram provided for an embodiment of the present invention.

[0058] Figure 11 This is a scaling effect diagram provided for an embodiment of the present invention.

[0059] Figure 12 This is a split-screen effect diagram provided in an embodiment of the present invention.

[0060] Figure 13 This is a blurred effect diagram provided for an embodiment of the present invention.

[0061] Figure 14 This is a light-emitting effect diagram provided for an embodiment of the present invention.

[0062] Figure 15 This is an example of the effect after adding text, provided in an embodiment of the present invention.

[0063] Figure 16 This is a rolling effect diagram of a film with a displacement fault, provided as an embodiment of the present invention.

[0064] Figure 17 This is a block diagram of an image processing device provided in an embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0066] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0067] As described in the background section, with the development of computer technology, smart terminals have been widely used. Users can use smart terminals to take photos or videos. To meet users' increasingly personalized needs for photos, adding special effects to photos or videos has become a popular way to beautify images. However, current special effects processing still needs to be improved, which is not conducive to increasing user engagement.

[0068] The image processing method, apparatus, electronic device, and readable storage medium provided in this invention acquire an image frame to be processed and a target permutation map corresponding to the image frame. Based on the displacement offset stored in the color channel of each first pixel in the target permutation map, the pixel displacement value of each second pixel in the image frame is determined. Then, based on the pixel displacement value of each second pixel, the sampling pixel corresponding to each second pixel in the image frame is determined. Finally, the pixel information of each second pixel is replaced with the pixel information of the sampling pixel corresponding to each second pixel. This achieves a fault effect with displacement, enriching the visual experience and improving user engagement. Furthermore, by storing the displacement offset in the color channel of the pixels in the permutation map and obtaining the pixel displacement value of each pixel in the image frame based on the displacement offset, the operation is simple and the response speed is fast, further enhancing user engagement.

[0069] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of the electronic device provided in this embodiment. Figure 1 As shown, the electronic device 100 includes an image processing device 10, a memory 20, a processor 30, and a communication unit 40. The memory 20, processor 30, and communication unit 40 are electrically connected directly or indirectly to each other to achieve signal transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0070] In this embodiment, the image processing apparatus 10 includes at least one software function module that can be stored in the memory 20 in the form of software or firmware. The processor 30 is used to execute the software function module stored in the memory 20. When the electronic device 100 is running, the processor 30 communicates with the memory 20 via a bus, and the processor 30 executes the software function module to implement the image processing method described in this embodiment.

[0071] The memory 20 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0072] Processor 30 is used to perform one or more functions described in this embodiment. In some embodiments, processor 30 may include one or more processing cores (e.g., a single-core processor (S) or a multi-core processor (S)). By way of example only, processor 30 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computing (RISC) computer, or a microprocessor, or any combination thereof.

[0073] For ease of explanation, only one processor is described in the electronic device 100. However, it should be noted that the electronic device 100 in this embodiment may also include multiple processors, and therefore the steps performed by one processor as described in this embodiment may also be performed jointly or individually by multiple processors. For example, if the processor of the electronic device performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, one processor performs step A, and a second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0074] In this embodiment, the process definition method disclosed in any implementation can be applied to the processor 30, or implemented by the processor 30.

[0075] The communication unit 40 is used to establish a communication connection between the electronic device 100 and other devices via a network, and to send and receive data via the network.

[0076] In some implementations, the network can be any type of wired or wireless network, or a combination thereof. By way of example only, the network may include wired networks, wireless networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, or near field communication (NFC) networks, or any combination thereof.

[0077] In this embodiment, the electronic device can be a terminal, laptop, tablet, ultra-mobile personal computer (UMPC), personal digital assistant (PDA), physical server, etc. This embodiment does not impose any restrictions on the specific type of electronic device.

[0078] Understandably, Figure 1 The structure shown is for illustrative purposes only. Electronic devices may also have more advanced features. Figure 1 Showing more or fewer components, or having with Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0079] based on Figure 1 The implementation architecture of this embodiment provides an image processing method, which is based on... Figure 1 The electronic device shown performs the following detailed explanation of the steps of the image processing method provided in this embodiment. Please refer to the references provided. Figure 2 The image processing method provided in this embodiment includes steps S101 to S104.

[0080] Step S101: Obtain the image frame to be processed and the target permutation map corresponding to the image frame.

[0081] This process can involve extracting video frames from a video, using these frames sequentially as image frames for adding effects. The video can be a captured video or an animated video generated from images. In essence, extracting frames from a video yields each image frame and its frame number.

[0082] After obtaining the image frame to be processed, the pixels in the image frame are mapped to the texture coordinate system. In this way, each pixel in the image frame has a position information (texture coordinate) in the texture coordinate system. For each pixel, the pixel information, i.e. the pixel value, can be obtained using this position information.

[0083] In this embodiment, the color channel of the pixel in the displacement map stores the displacement offset, which is used to determine the pixel position offset information of the pixel in the image frame to be processed, that is, the position offset information of the pixel in the image frame in the texture coordinate system.

[0084] In this embodiment, the displacement offset ranges from -1 to 1, but the texture coordinate system ranges from 0 to 1. Therefore, this embodiment converts the displacement offset from -1 to 1 to 0 to 1 and stores it in the color channel of the pixel in the displacement map. That is, the displacement offset in the color channel of the pixel in the displacement map is 0 to 1, such as... Figure 3 As shown.

[0085] Understandably, since a pixel has multiple color channels, this embodiment can store different displacement offsets in different color channels. For example, for a pixel's RGBA color channels, the R color channel stores displacement offset 'a', the G color channel stores displacement offset 'b', the B color channel stores displacement offset 'c', and the A color channel stores displacement offset 'd'. Understandably, in the same permutation map, different pixels can have different displacement offsets stored in their color channels.

[0086] In an optional implementation, the permutation map can be stored in the electronic device in advance. After the image frame to be processed is obtained, the stored permutation map can be used as the target permutation map corresponding to the image frame.

[0087] In an optional implementation, the target permutation map corresponding to the image frame can be obtained based on the image type of the image frame. For example, multiple permutation maps can be set, each corresponding to an image type, such as permutation map a corresponding to a person, permutation map b corresponding to a landscape, permutation map c corresponding to an animal, etc. After obtaining the image frame, the image frame can be identified to obtain the image type of the image frame. Assuming the image type is a person, then permutation map a is the target permutation map corresponding to the image frame. Understandably, the image type of the image frame can be determined using methods such as classification neural networks, which will not be elaborated on in detail here.

[0088] In this embodiment, to obtain the pixel position offset information of each pixel in the image frame, a target permutation map corresponding to the image frame can also be obtained based on the resolution of the image frame, that is, the resolution of the target permutation map is the same as the resolution of the image frame. For example, multiple permutation maps can be set, each permutation map corresponding to an image resolution. After obtaining the image frame, the resolution of the image frame can be compared with the resolution of each permutation map to obtain a target permutation map with the same resolution as the image frame.

[0089] After obtaining the target permutation map corresponding to the image frame, the pixel position offset information of the pixels in the image frame can be determined based on the displacement offset in the color channel of the pixels in the target permutation map. It should be noted that, in order to distinguish between the pixels in the image frame and the pixels in the permutation map, the pixels in the permutation map will be described as the first pixel, and the pixels in the image frame will be described as the second pixel.

[0090] Step S102: Determine the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channel of each first pixel.

[0091] The pixel displacement value refers to the pixel position offset information, that is, the value of displacement required for the second pixel to be in its original position.

[0092] In this embodiment, when determining the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channel of each first pixel, the displacement offset of one color channel from the color channels of each first pixel can be selected to determine the pixel displacement value of each second pixel in the image frame. Alternatively, the displacement offsets of at least two color channels from the color channels of each first pixel can be selected to determine the pixel displacement value of each second pixel in the image frame. For example, the displacement offset of any one of the R, G, B, and A color channels can be selected to determine the pixel displacement value of each second pixel in the image frame. Alternatively, the displacement offsets of at least two of the R, G, B, and A color channels can be selected to determine the pixel displacement value of each second pixel in the image frame.

[0093] Since the position information of the second pixel in the texture coordinate system includes both horizontal and vertical position information, when selecting the displacement offset of a color channel to determine the pixel displacement value of the second pixel, the pixel displacement value in the horizontal direction and the pixel displacement value in the vertical direction can be determined simultaneously based on the displacement offset of that color channel.

[0094] When selecting the displacement offsets of multiple color channels, the pixel displacement value of the second pixel in different directions can be determined based on these displacement offsets. For example, for the R and G color channels, the displacement offset of the R color channel can be used to determine the pixel displacement value of the second pixel in the horizontal direction, and the displacement offset of the G color channel can be used to determine the pixel displacement value of the second pixel in the vertical direction. Similarly, for the R, G, and B color channels, the average displacement offset of the R and G color channels can be used to determine the pixel displacement value of the second pixel in the horizontal direction, and the displacement offset of the B color channel can be used to determine the pixel displacement value of the second pixel in the vertical direction. Likewise, for the R, G, B, and A color channels, the average displacement offset of the R and G color channels can be used to determine the pixel displacement value of the second pixel in the horizontal direction, and the average displacement offset of the B and A color channels can be used to determine the pixel displacement value of the second pixel in the vertical direction.

[0095] For ease of calculation, this embodiment uses the displacement offset of one color channel to determine the pixel displacement value of the second pixel in the horizontal direction and the pixel displacement value in the vertical direction. It can be understood that when determining the pixel displacement value of the second pixel using the displacement offset of one color channel, the color channel selected for each first pixel is the same, such as selecting the displacement offset of each first pixel in the R color channel.

[0096] After determining the selected color channel, the pixel displacement value of each second pixel in the image frame can be determined based on the displacement offset in the selected color channel of each first pixel.

[0097] In an optional implementation, after obtaining the target displacement map, it can also be mapped to a texture coordinate system. Thus, each pixel in the target displacement map has positional information in the texture coordinate system. For each pixel, this positional information can be used to obtain the displacement offset in its color channel. Based on this, when determining the pixel displacement value of each second pixel in the image frame according to the displacement offset in the color channel of each first pixel, for each second pixel in the image frame, based on its positional information, it is searched whether there exists a first pixel in the target displacement map with the same positional information as that second pixel. If such a first pixel exists, the displacement offset in its color channel of the first pixel with the same positional information is used as the pixel displacement value of that second pixel. If not, the target displacement map is searched for the first pixel closest to the positional information of that second pixel, and the displacement offset in its color channel is used as the pixel displacement value of that second pixel. If multiple closest first pixels exist simultaneously, the average value of these multiple first pixels is used as the pixel displacement value of that second pixel.

[0098] It should be noted that since the displacement offset ranges from -1 to 1, while the displacement offset stored in the color channels of the pixels in the target permutation map ranges from 0 to 1, when determining the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channels of each first pixel, it is necessary to convert the displacement offset from the color channels to -1 to 1. Optionally, the displacement offset in the color channels can be converted using the following formula.

[0099] DisplacePercent=DisplacementMapColor.r*2.0-1.0

[0100] Where DisplacePercent is the displacement offset obtained after converting the displacement offset in the color channel, and DisplacementMapColor.r is the displacement offset in the R color channel. It can be understood that the displacement offset can also be obtained from any other color channel, not just the R color channel.

[0101] Step S103: Determine the sampling pixel points in the image frame corresponding to each second pixel point based on the pixel displacement value of each second pixel point.

[0102] The pixel displacement value of the second pixel includes the pixel displacement value in the horizontal direction and the pixel displacement value in the vertical direction.

[0103] When determining the sampling pixel corresponding to each second pixel in the image frame based on the pixel displacement value of each second pixel, the sampling position of the second pixel can be obtained by adding the pixel displacement value of the second pixel to its position information in both the horizontal and vertical directions. Specifically, this can be achieved using the following code:

[0104] vec2 new_texcoord=vec2(v_texcoord.x+percentX, v_texcoord.y+percentY)

[0105] Where vec2 new_texcoord is the sampling position, v_texcoord.x is the position information of the second pixel in the horizontal direction, v_texcoord.y is the position information of the second pixel in the vertical direction, percentX is the pixel displacement value of the second pixel in the horizontal direction, and percentY is the pixel displacement value of the second pixel in the vertical direction.

[0106] After obtaining the sampling position of each second pixel, for each second pixel, the second pixel corresponding to the sampling position of that second pixel in the image frame is taken as the sampling pixel.

[0107] Step S104: Replace the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel.

[0108] After obtaining the sampled pixels corresponding to each second pixel, for each second pixel, texture pixel sampling can be performed on that second pixel based on the sampled pixels corresponding to it. This means replacing the pixel information of the second pixel with the pixel information of the sampled pixels corresponding to it. For example, for second pixel A, whose corresponding sampled pixel is pixel B, assuming the pixel information of second pixel A is Y1 and the pixel information of pixel B is Y2, after replacing the pixel information of second pixel A with the pixel information of pixel B, the pixel information of second pixel A becomes Y2.

[0109] Understandably, when the displacement offset in the color channel of the first pixel corresponding to a second pixel in an image frame is 0, the pixel displacement value of the second pixel is also 0. Therefore, the sampled pixel corresponding to the second pixel is itself, meaning the pixel information of the second pixel does not change. Based on this, after replacing the pixel information, this embodiment can produce an image frame with some areas showing a displacement fault effect and some areas without. The image processing method provided in this embodiment acquires the image frame to be processed and the target permutation map corresponding to the image frame. Based on the displacement offset stored in the color channel of each first pixel in the target permutation map, the pixel displacement value of each second pixel in the image frame is determined. Then, based on the pixel displacement value of each second pixel, the sampling pixel corresponding to each second pixel in the image frame is determined. After that, the pixel information of each second pixel is replaced with the pixel information of the sampling pixel corresponding to each second pixel. In this way, a fault effect with displacement can be achieved, enriching the visual experience and improving user stickiness. Furthermore, by storing the displacement offset in the color channel of the pixel in the permutation map and obtaining the pixel displacement value of each pixel in the image frame based on the displacement offset, the operation is simple and the response speed is fast, further improving user stickiness.

[0110] To ensure that each second pixel in the image frame is fitted with an accurate displacement offset, in this embodiment, the target permutation map is adapted to the image frame to be processed; that is, the target permutation map and the image frame to be processed have the same resolution. Thus, for each second pixel in the image frame, a matching first pixel can be found in the target permutation map.

[0111] Furthermore, considering that image frames have various resolutions in practical applications, if a corresponding adaptation permutation map were to be set for each resolution, multiple permutation maps would be required, consuming resources. To reduce resource consumption, in this embodiment, a single permutation map can be adapted to image frames of different resolutions through cropping. Based on this, in this embodiment, the step of obtaining the target permutation map corresponding to an image frame may include:

[0112] (a) Obtain the initial permutation map.

[0113] (b) Obtain the aspect ratio of the image frame based on its resolution.

[0114] (c) Based on the aspect ratio of the image frame, the initial permutation map is cropped to obtain the target permutation map.

[0115] The initial displacement map is an image pre-imported and stored in the electronic device. Each first pixel in the initial displacement map stores a pre-set displacement offset in its color channel. To adapt to image frames of different resolutions, in this embodiment, the initial displacement map can be a high-resolution image; optionally, the resolution of the initial displacement map in this embodiment can be 1280*1280.

[0116] In this embodiment, the aspect ratio of the image frame is obtained based on its resolution. This can be achieved by comparing the horizontal resolution to the vertical resolution of the image frame, and then using the ratio as the aspect ratio. For example, if the resolution of the image frame is 720*1280, where 720 is the horizontal resolution and 1280 is the vertical resolution, then the aspect ratio is 720 / 1280 = 9:16. Alternatively, if the resolution of the image frame is 960*1280, where 960 is the horizontal resolution and 1280 is the vertical resolution, then the aspect ratio is 960 / 1280 = 3:4.

[0117] After obtaining the aspect ratio of the image frame, the initial permutation map can be cropped according to the aspect ratio to obtain the target permutation map.

[0118] Optionally, the first pixel of the initial displacement map can be mapped to a texture coordinate system first. Then, the aspect ratio of the image frame is compared with several set ratio values. Based on the comparison results, the new position information (texture coordinates) of each first pixel in the initial displacement map is determined. Afterward, based on the new position information (texture coordinates) of each first pixel in the initial displacement map, the initial displacement map is cropped (sampled) to obtain the target displacement map. The specific implementation can be shown in the following code:

[0119]

[0120]

[0121] Where, ratio is the image aspect ratio; image_width is the horizontal resolution of the image frame; image_height is the vertical resolution of the image frame; delta is the error; vec2 new_texcoord is the position information (texture coordinates) of the first pixel in the initial permutation map; uv is the texture coordinate in the texture coordinate system; new_texcoord-=vec2(0.5) means shifting the position information of the first pixel in the initial permutation map by 0.5 to shift the center point of the initial permutation map to the origin of the texture coordinate system, new_texcoord+=vec2(0.5) means shifting the position information of the first pixel by 0.5 to restore it to the initial position; new_texcoord.x is the horizontal position information of the first pixel in the initial permutation map, and new_texcoord.y is the vertical position information of the first pixel in the initial permutation map.

[0122] Based on the above code, when the image aspect ratio <= 0.5625 + delta, the horizontal position information is updated as: new_texcoord.x = UV (original texture coordinates) * (720.0 / 1280.0), while the vertical position information remains unchanged. Thus, when the image aspect ratio <= 0.5625 + delta, the new position information of each first pixel in the initial permutation map is (new_texcoord.x = UV * (720.0 / 1280.0), new_texcoord.y = UV).

[0123] When the image aspect ratio <= 0.75 + delta, the horizontal position information is updated as: new_texcoord.x = UV(original texture coordinates) * (720.0 / 1280.0), and the vertical position information is updated as: new_texcoord.y = UV(original texture coordinates) * (960.0 / 1280.0). Thus, when the image aspect ratio <= 0.75 + delta, the new position information of each first pixel in the initial permutation map is (new_texcoord.x = UV * (720.0 / 1280.0), new_texcoord.y = UV * (960.0 / 1280.0)).

[0124] When the image aspect ratio <= 1.0 + delta, the position information in the horizontal direction is updated as: new_texcoord.x = UV(original texture coordinates) * (720.0 / 1280.0), and the position information in the vertical direction is updated as: new_texcoord.y = UV(original texture coordinates) * (720.0 / 1280.0). Thus, when the image aspect ratio <= 1.0 + delta, the new position information of each first pixel in the initial permutation map is (new_texcoord.x = UV * (720.0 / 1280.0), new_texcoord.y = UV * (720.0 / 1280.0)).

[0125] When the image aspect ratio <= 1.333 + delta, the position information in the horizontal direction is updated as: new_texcoord.x = UV(original texture coordinates) * (960.0 / 1280.0), and the position information in the vertical direction is updated as: new_texcoord.y = UV(original texture coordinates) * (720.0 / 1280.0). Thus, when the image aspect ratio <= 1.333 + delta, the new position information of each first pixel in the initial permutation map is (new_texcoord.x = UV * (960.0 / 1280.0), new_texcoord.y = UV * (720.0 / 1280.0)).

[0126] When the image aspect ratio is otherwise (e.g., 16 / 9), the horizontal position information remains unchanged, while the vertical position information is updated as follows: new_texcoord.y = UV(original texture coordinates) * (720.0 / 1280.0).

[0127] Thus, when the image aspect ratio is otherwise, the new position information of each first pixel in the initial permutation map is (new_texcoord.x = UV, new_texcoord.y = UV * (720.0 / 1280.0)).

[0128] After obtaining the new position information of each first pixel in the initial permutation map, the initial permutation map is cropped (sampled) based on the new position information to obtain the target permutation map, such as... Figure 4 As shown, when the image frame resolution is 720.0*1280.0 and the image aspect ratio is 0.5625, Figure 4 The area within the box represents the cropped image result, which is the cropped target permutation map. The cropped target permutation map is then adapted to the resolution of the image frame (720.0*1280.0).

[0129] This embodiment crops the initial permutation map by adjusting the image frame resolution, making the initial permutation map applicable to image frames with various aspect ratios, thus reducing resource consumption. Figure 5 As shown, a 1280*1280 resolution displacement map, after being cropped, can be applied to image frames with five different aspect ratios (9:16, 16:9, 3:4, 4:3, 1:1).

[0130] To ensure that each image frame has a different random fault effect, this embodiment uses a permutation map sequence frame approach, where each image frame has a different displacement offset, thus achieving different random displacement fault effects across different image frames. Based on this, the step of obtaining the initial permutation map in this embodiment may further include:

[0131] Based on the frame number of the image frame, determine the initial permutation map in the preset permutation map sequence that matches the frame number of the image frame.

[0132] In this embodiment, the permutation graph sequence frame contains multiple permutation graphs, and each permutation graph has a unique frame number according to the sorting order. For example, if there are n permutation graphs in the permutation graph sequence frame, and they are sorted in the order {Y1, Y2, Y3...Yn}, then the set of frame numbers can be represented as {1, 2, 3...n}, that is, the frame number of permutation graph Y1 is the 1st frame, the frame number of permutation graph Y2 is the 2nd frame, the frame number of permutation graph Y3 is the 3rd frame, ..., the frame number of permutation graph Y2 is the 2nd frame, and the frame number of permutation graph Y3 is the nth frame.

[0133] In this embodiment, the displacement offset stored in the color channel of the first pixel in each permutation map sequence frame is different. For example, the displacement offset stored in the color channel of each permutation map in the permutation map sequence frame for the same first pixel position may also be different.

[0134] In this embodiment, the initial permutation map that matches the frame number of the image frame in the preset permutation map sequence is determined based on the frame number of the image frame. This can be achieved by comparing the frame number of the image frame with the frame number of each permutation map in the preset permutation map sequence to obtain the initial permutation map with matching frame numbers.

[0135] Optionally, the frame number of the image frame can be compared with the frame number of each permutation map in the preset permutation map sequence. If a permutation map with the same frame number as the image frame is obtained, it is used as the initial permutation image. If no permutation map with the same frame number as the image frame is obtained, the frame number of the image frame is subtracted from the maximum frame number in the preset permutation map sequence. The difference is then compared with the frame number of each permutation map in the preset permutation map sequence. If a permutation map with the same frame number as the difference is obtained, it is used as the initial permutation image. If no permutation map with the same frame number as the difference is obtained, the difference is subtracted from the maximum frame number in the preset permutation map sequence, and this process is repeated until a permutation map with the same frame number as the difference is obtained, which is then used as the initial permutation image.

[0136] For example, if the frame numbers of the permutations in the preset permutation map sequence are {1, 2, 3, 4}, and the frame number of the image frame is frame 3, then the permutation map corresponding to frame 3 in the preset permutation map sequence is used as the initial permutation map. When the frame number of the image frame is frame 6, then the frame number of the image frame is subtracted from the maximum frame number in the preset permutation map sequence, that is, the difference between frame 6 and frame 4 is taken, and the difference is 2. After comparing the difference with the frame number of each permutation map in the preset permutation map sequence, the permutation map with frame number 2 can be used as the initial permutation map.

[0137] After obtaining an initial permutation map whose frame number matches the frame number of the image frame, the initial permutation map is cropped according to the aspect ratio of the image frame to obtain the target permutation map corresponding to the image frame.

[0138] In this embodiment, since the displacement offset stored in the color channel of the first pixel in each permutation map sequence frame is different, by determining the permutation map corresponding to the image frame according to the frame number of the image frame, each image frame can have a corresponding permutation map with different displacement offsets, thus achieving different random displacement fault effects for different image frames.

[0139] Considering that if the entire image frame is shifted by a significant value, the original image may become unrecognizable, please refer to the relevant documentation. Figure 6 In this embodiment, the step of determining the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channel of each first pixel may further include steps S201 to S203.

[0140] Step S201: Obtain the offset coefficient of each second pixel in the image frame according to the preset offset threshold and scaling factor.

[0141] Step S202: For each second pixel in the image frame, based on the position information of the second pixel, obtain the target first pixel in the target permutation map whose position information matches that of the second pixel.

[0142] Step S203: Obtain the pixel displacement value of the second pixel based on the displacement offset and offset coefficient in the color channel of the first pixel.

[0143] The offset threshold includes both a horizontal offset threshold and a vertical offset threshold. Optionally, in this embodiment, the horizontal offset threshold can be 5 pixels, and the vertical offset threshold can be 2 pixels. After normalization, the horizontal offset threshold is 5 / image width, and the vertical offset threshold is 2 / image height. For example, when the image width and height are 720*1280, the horizontal offset threshold is 5 / 720, and the vertical offset threshold is 2 / 1280.

[0144] In this embodiment, the scaling factor is the scaling ratio of the offset, which can be set based on actual needs and is not limited in specific terms. For example, it can be 10%, 15%, etc.

[0145] Optionally, the offset coefficient of each second pixel in the image frame can be obtained according to the preset offset threshold and scaling factor. This can be achieved by multiplying the offset threshold and scaling factor and using the product as the offset coefficient.

[0146] Optionally, the offset coefficient of each second pixel in the image frame can be obtained according to the preset offset threshold and scaling factor. This can be achieved by adding or subtracting the offset threshold and scaling factor, and using the result of the addition or subtraction as the offset coefficient.

[0147] Understandably, since the offset threshold includes both a horizontal offset threshold and a vertical offset threshold, when obtaining the offset coefficient based on the offset threshold and the scaling factor, the horizontal offset coefficient can be obtained from the horizontal offset threshold and the scaling factor, and the vertical offset coefficient can be obtained from the vertical offset threshold and the scaling factor. Therefore, the offset coefficient in this embodiment also includes both a horizontal offset coefficient and a vertical offset coefficient.

[0148] Since the resolution of the target permutation map obtained by cropping is adapted to the resolution of the image frame, after mapping the target permutation map and the pixels of the image frame to the texture coordinate system, for each second pixel in the image frame, a first pixel whose position information is the same as that of the second pixel can be found in the target permutation map based on the position information of the second pixel in the texture coordinate system. Therefore, in this embodiment, for each second pixel in the image frame, obtaining the target first pixel whose position information matches that of the second pixel in the target permutation map based on the position information of the second pixel can be done by finding the first pixel in the target permutation map whose position information is the same as that of the second pixel based on the position information of the second pixel, and taking the first pixel with the same position as the target first pixel.

[0149] For each second pixel in an image frame, the pixel displacement value of the second pixel is obtained based on the displacement offset and offset coefficient in the color channel of the target first pixel corresponding to the second pixel. Alternatively, the sum of the displacement offset and the offset coefficient can be used as the pixel displacement value of the second pixel.

[0150] In order to achieve a slight offset effect, in this embodiment, the pixel displacement value of the second pixel is obtained based on the displacement offset and offset coefficient in the color channel of the target first pixel corresponding to the second pixel. This can be achieved by multiplying the displacement offset and the offset coefficient, and using the multiplied value as the pixel displacement value of the second pixel.

[0151] Understandably, the offset coefficient includes both horizontal and vertical offset coefficients. Therefore, multiplying the horizontal offset coefficient by the displacement offset in the color channel yields the pixel displacement value in the horizontal direction, and multiplying the vertical offset coefficient by the displacement offset in the color channel yields the pixel displacement value in the vertical direction. Specifically, this can be obtained using the following formula:

[0152] percentX=DisplacePercent*offset.x

[0153] percentY=DisplacePercent*offset.y

[0154] Where DisplacePercent is the displacement offset, percentX is the pixel displacement value of the second pixel in the horizontal direction, percentY is the pixel displacement value of the second pixel in the vertical direction, offset.x is the offset coefficient in the horizontal direction, and offset.y is the offset coefficient in the vertical direction.

[0155] After obtaining the pixel displacement value of each second pixel in the image frame, the corresponding sampling pixel in the image frame can be determined based on the pixel displacement value. Then, the pixel information of each second pixel is replaced with the pixel information of the corresponding sampling pixel. Since the pixel displacement value is obtained by multiplying the displacement offset by the offset coefficient, pixel sampling based on the pixel displacement offset results in a slight displacement effect in the final image frame, such as... Figure 7 As shown.

[0156] In an optional implementation, to make the fault effect smoother, in this embodiment, after replacing the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel to obtain the replaced image frame, the replaced image frame can be mixed with the unreplaced image frame to make the displacement effect smoother, such as... Figure 8 As shown.

[0157] The image processing method provided in this embodiment stores displacement offsets in the color channels of pixels in the displacement map, obtains pixel displacement values ​​of each pixel in the image frame based on the displacement offsets, and samples each pixel based on the pixel displacement values ​​of each pixel, thereby achieving a fault effect similar to displacement. It is simple to operate, has a fast response speed, and improves user stickiness.

[0158] To further enrich the special effects and enhance user engagement, this embodiment can also perform color shifting on the image frames to achieve color shifting effects. Please refer to [the relevant documentation / reference]. Figure 9 After acquiring the image frame to be processed, the image processing method provided in this embodiment further includes steps S301 to S302.

[0159] Step S301: Obtain the color information of each second pixel in each color channel and the color offset of each color channel.

[0160] Step S302: For each color channel, based on the color offset of the color channel, determine the color offset pixel corresponding to each second pixel in the image frame, and replace the color information of each second pixel in the color channel with the color information of the color offset pixel corresponding to each second pixel in the color channel.

[0161] The color information of the second pixel in each color channel includes the color value of the second pixel in the R color channel, G color channel, B color channel and the transparency in the A color channel.

[0162] In this embodiment, the color offset of a color channel refers to the displacement offset of a pixel for that color channel. It can be set according to actual needs and is not limited in specific terms. The color offset of each color channel can be the same or different.

[0163] Optionally, in this embodiment, only the R color channel and the B color channel are color shifted. Therefore, in this embodiment, the color shift amount of the G color channel and the A color channel is set to 0, and the color shift amount of the R color channel and the B color channel is set to a value greater than 0. The color shift amounts of the R color channel and the B color channel are different.

[0164] In this embodiment, for each color channel, the color offset pixel corresponding to each second pixel in the image frame is determined according to the color offset of the color channel. This can be achieved by adding the color offset of the color channel to the position information of each second pixel to obtain the color offset position of each second pixel, and then using the pixel corresponding to the color offset position of each second pixel as the color offset pixel corresponding to each second pixel in that color channel.

[0165] Specifically, a color offset is added to the position information of the second pixel, including adding a color offset to the position information in the horizontal direction and adding a color offset to the position information in the vertical direction.

[0166] For example, if the color offset of the R color channel is set to colorROffset and the color offset of the B color channel is set to colorBOffset, then for each second pixel in the R color channel, the color offset position of that second pixel is (v_texcoord.x + colorROffset, v_texcoord.y + colorROffset). The pixel corresponding to (v_texcoord.x + colorROffset, v_texcoord.y + colorROffset) in the image frame is the color offset pixel corresponding to that second pixel in the R color channel.

[0167] For the B color channel, for each second pixel, the color offset position of the second pixel is (v_texcoord.x+colorBOffset, v_texcoord.y+colorBOffset). The pixel corresponding to (v_texcoord.x+colorBOffset, v_texcoord.y+colorBOffset) in the image frame is the color offset pixel corresponding to the second pixel in the B color channel.

[0168] Where v_texcoord.x represents the position information of the second pixel in the horizontal direction, and v_texcoord.y represents the position information of the second pixel in the vertical direction.

[0169] In an optional implementation, a color offset is added to the position information of the second pixel. The color offset can be added to the position information in only one direction (horizontal or vertical), while the position information in the other direction remains unchanged.

[0170] After obtaining the color offset pixel, the color information of the second pixel in that color channel can be replaced with the color information of the color offset pixel in that color channel. For example, for the R color channel, for each second pixel in the image frame, after obtaining the color offset pixel corresponding to that second pixel in the R color channel, the color information of that second pixel in the R color channel can be replaced with the color information of the corresponding color offset pixel in the R color channel. The same applies to the B color channel, and will not be elaborated here.

[0171] In an optional implementation, a time parameter can be added to the color offset of each color channel. This involves multiplying the color offset by the time parameter, thus achieving a color shift effect that changes over time when playing image frames. Optionally, the time parameter can be max(sin(time*(PI / duration)),0.0), where duration is the time period of the color shift effect, pi is the mathematical constant pi, and time is the current time.

[0172] The image processing method provided in this embodiment achieves the color shift effect of the image frame by shifting the color of each second pixel point through the color shift amount of each color channel. Figure 10 As shown.

[0173] To further enrich the special effects and increase user engagement, this embodiment can also add borders to the image frames to achieve border effects.

[0174] Optionally, the image frame can be centered and scaled first, and then a border can be added to the scaled image frame to ensure that the size of the image frame after adding the border does not exceed the original size.

[0175] Centering and scaling of image frames can be achieved using the following code:

[0176] vec2 texcoordTemp = v_texcoord - 0.5; / / Translate by 0.5, moving the image center point to the origin;

[0177] texcoordTemp* = 1.0 / 0.9; / / Multiply by 1.0 / 0.9;

[0178] texcoordTemp = texcoordTemp + 0.5; / / After scaling, translate by 0.5 to restore the initial position.

[0179] Where v_texcoord is the initial position of the pixel in the image frame, and v_texcoord-0.5 means that the pixel in the image frame is translated by 0.5 to move the center point of the image to the origin.

[0180] `texcoordTemp* = 1.0 / 0.9` means multiplying the translated pixel position information by 1.0 / 0.9 to scale the image. `texcoordTemp = texcoordTemp + 0.5` means shifting the scaled position information by 0.5 to restore it to its initial position. Figure 11 As shown, Figure 11 This is the effect after scaling down. The red area is the background area, and you can add borders to the image frame in the background area.

[0181] To further enrich the special effects and enhance user engagement, this embodiment can also perform three-screen processing on the image frames to achieve split-screen effects. Based on this, the image processing method provided in this embodiment also includes:

[0182] (1) For each second pixel in the image frame, if the position information of the second pixel in the first direction is less than the first preset value, then the target pixel corresponding to the second pixel is determined according to the sum of the preset displacement value and the position information, and the pixel information of the second pixel is replaced with the pixel information of the target pixel.

[0183] (2) If the position information of the second pixel in the first direction is greater than the second preset value, then the target pixel corresponding to the second pixel is determined according to the difference between the position information and the preset displacement value, and the pixel information of the second pixel is replaced with the pixel information of the target pixel.

[0184] The first direction can be either horizontal or vertical. To achieve a split-screen effect that is vertically split, the first direction is vertical; to achieve a split-screen effect that is horizontally split, the first direction is horizontal.

[0185] The first preset value and the second preset value refer to the screen splitting threshold based on the three-screen setting. If the three screens are evenly divided, the first preset value and the second preset value can be three equal division points. If the three screens are not evenly divided, the first preset value and the second preset value can be set based on the screen splitting requirements, without any specific restrictions. The second preset value is greater than the first preset value.

[0186] In this embodiment, the displacement value is a pixel displacement value set according to the screen splitting requirements. For each second pixel in the image frame, if the position information of the second pixel in the first direction is less than a first preset value, the screen splitting position of the second pixel is obtained according to the sum of the preset displacement value and the position information. The target pixel corresponding to the screen splitting position in the image frame can then be determined. If the position information of the second pixel in the first direction is greater than a second preset value, the screen splitting position of the second pixel is obtained according to the difference between the position information and the preset displacement value, i.e., the difference obtained by subtracting the displacement value from the position information. The target pixel corresponding to the screen splitting position in the image frame can then be determined.

[0187] After obtaining the target pixel of the second pixel, the pixel information of the second pixel can be replaced with the pixel information of the target pixel.

[0188] It should be noted that if the position information of the second pixel in the first direction is neither less than the first preset value nor greater than the second preset value, then the second pixel will not be processed, that is, its pixel information will not be replaced. Thus, for an image frame, a split-screen effect can be achieved where the middle part (the part whose position information is between the first and second preset values) is displayed in the center, the upper part (the part whose position information is less than the first preset value) displays the latter half of the image, and the lower part (the part whose position information is greater than the second preset value) displays the upper part of the image. Figure 12 As shown.

[0189] To enrich the special effects, this embodiment can also add a shaking effect during the screen splitting process. Based on this, the step of determining the target pixel corresponding to the second pixel based on the sum of the preset displacement value and position information can further include:

[0190] Based on the frame number of the image frame, a target jitter value whose ordinal number matches the frame number is determined from a preset jitter value sequence; the jitter value sequence includes multiple jitter values, each with a unique ordinal number.

[0191] The target pixel corresponding to the second pixel is obtained by summing the displacement value, the target jitter value, and the position information.

[0192] Accordingly, the step of determining the target pixel corresponding to the second pixel based on the difference between the position information and the displacement value may further include:

[0193] The target pixel corresponding to the second pixel is obtained based on the difference between the position information, displacement value, and target jitter value.

[0194] The jitter value sequence includes multiple jitter values, each with a unique ordinal number. Determining the target jitter value corresponding to the ordinal number and frame ordinal number in the preset jitter value sequence can be done by referring to the process of determining the initial permutation map that matches the frame ordinal number in the preset permutation map sequence with the frame ordinal number of the image frame based on the frame ordinal number of the image frame; this process will not be elaborated upon here.

[0195] This embodiment determines the target jitter value that matches the frame number in the preset jitter value sequence based on the frame number of the image frame. This allows each frame to have a different jitter effect, further enriching the special effects and increasing user engagement.

[0196] After obtaining the target jitter value, for a second pixel whose position information in the first direction is less than a first preset value, the target pixel corresponding to the second pixel can be obtained based on the sum of the preset displacement value, the target jitter value, and the position information. For a second pixel whose position information in the first direction is greater than the second preset value, the target pixel corresponding to the second pixel can be obtained based on the difference between the position information, the preset displacement value, and the target jitter value, i.e., the difference between the position information, the displacement value, and the target jitter value. Optionally, when the first direction is a vertical direction, the corresponding target pixel can be obtained using the following code:

[0197]

[0198] Where (texcoordTemp.x, texcoordTemp.y) and uv are the position information of the second pixel, texcoordTemp.y and uv.y are the position information of the second pixel in the vertical direction, yProgress is the target jitter value corresponding to the image frame, k1 is the first preset value, k2 is the second preset value, m is the displacement value, and m = k2.

[0199] To further enrich the effects and enhance user engagement, this embodiment can also blur the image frames to achieve a slight motion blur effect. Based on this, the image processing method provided in this embodiment may further include:

[0200] (a) For each second pixel in the image frame, the second pixel is sampled in the second direction according to the preset blur radius to obtain the first sampled pixel.

[0201] (b) Based on the blur radius, sample each first sampling pixel in the second direction to obtain the second sampling pixel.

[0202] (c) Obtain the target pixel information of the second pixel based on the average value of the pixel information of each first sampled pixel and each second sampled pixel.

[0203] The blur radius can be set based on the desired blurring effect and is not specifically limited. In this embodiment, to achieve a slight blurring effect, the blur radius can be set to 1. The second direction can be a horizontal direction, a vertical direction, or a direction corresponding to other angles.

[0204] Optionally, in this embodiment, when the second direction is vertical, for each second pixel in the image frame, the second pixel is sampled in the second direction according to a preset blur radius to obtain a first sampled pixel. This can be based on the blur radius, by upsampling or downsampling the second pixel in the second direction to obtain the first sampled pixel. For example, when the blur radius is 1, upsampling or downsampling the second pixel in the second direction to obtain the first sampled pixel includes a pixel offset upwards by 1 pixel and a pixel offset downwards by 1 pixel.

[0205] Accordingly, based on the blur radius, each first sampling pixel is sampled in the second direction. This can be referred to as sampling the second pixel in the second direction based on the blur radius, which will not be elaborated here.

[0206] For each second pixel in the image frame, after obtaining all the corresponding first and second sampled pixels, the pixel information of each first and second sampled pixel is summed, and then the average value is calculated. This average value is used as the target pixel information for that second pixel. In this way, a slight directional blur effect can be added to the image frame. Figure 13 As shown.

[0207] To further enrich the special effects, this embodiment can also add a glowing layer to the image frame to simulate the light effect of film projection and add noise to simulate a vignetting effect with grainy noise. Based on this, after replacing the pixel information of each second pixel with the pixel information of the corresponding sampled pixel, the image processing method provided in this embodiment further includes:

[0208] (A) Based on the frame number of the image frame, determine the target luminous layer in the preset luminous foreground sequence frames whose frame number matches that of the image frame.

[0209] (B) Add the target glow layer to the image frame to obtain the image frame after adding the target glow layer;

[0210] (C) Add a preset noise layer to the image frame after adding the target glow layer.

[0211] The preset luminous foreground sequence frames include multiple luminous layers, each with a frame number.

[0212] Optionally, based on the frame number of the image frame, a target luminescent layer whose frame number in the preset luminescent foreground sequence frames matches the frame number of the image frame is determined. This can be achieved by comparing the frame number of the image frame with the frame number of each luminescent layer in the preset luminescent foreground sequence frames to obtain the target luminescent layer with the matching frame number. The specific process can refer to the process of determining the initial permutation map whose frame number in the preset permutation map sequence frames matches the frame number of the image frame based on the frame number of the image frame, which will not be elaborated here.

[0213] After obtaining the target glow layer, you can add the target glow layer to the image frame using the blending modes. This can be achieved using the code vec4 result2 = result1 + sprite_louguang_color, where sprite_louguang_color is the target glow layer, result1 is the image frame, and result2 is the image frame after adding the target glow layer.

[0214] This embodiment adds a glowing layer using an additive blending mode, which can achieve a film-like lighting effect, such as... Figure 14 As shown.

[0215] In this embodiment, a preset noise layer is added to the image frame after the target glow layer is added. This can be done using the code vec4. The function `result3 = vec4(result2.rgb * (1.0 - grain_noise_color.a) + grain_noise_color.rgb * grain_noise_color.a, 1.0)` is implemented where `grain_noise_color.a` represents the A color channel information of pixels in the noise layer, `grain_noise_color.rgb` represents the RGB color channel information of pixels in the noise layer, `result3` is the image frame after adding the noise layer, `result2.rgb` represents the RGB color channel information of pixels in the image frame after adding the target glow layer, and `result2.rgb * (1.0 - grain_noise_color.a) + grain_noise_color.rgb * grain_noise_color.a` represents the blending of the colors of the image frame after adding the target glow layer and the noise layer. `vec4(result2.rgb * (1.0 - grain_noise_color.a) + ...` indicates the mixing of the colors of the image frame after adding the target glow layer and the noise layer.

[0216] In `grain_noise_color.rgb*grain_noise_color.a, 1.0)`, the 1 represents the value of the A color channel of a pixel in the image frame after the target glow layer has been added.

[0217] This embodiment achieves a vignetting effect with a grainy noise profile by adding noise and vignetting material to the image frame.

[0218] In an optional implementation, to prevent image frame theft, this embodiment can also add information such as text to the image frame, such as... Figure 15 As shown, this can be achieved through the code vec4 result4 = result3 + logo_color, where logo_color is the text layer information and result4 is the image frame after adding text.

[0219] This embodiment achieves a film scrolling effect with displacement faults by processing each image frame with positional offset, color offset, split-screen jitter blurring, adding a glow layer and a noise layer, such as... Figure 16 As shown, this enriches the special effects and increases user engagement.

[0220] Based on the same inventive concept, please refer to Figure 17 This invention provides an image processing device 10, which is applied to... Figure 1 The electronic devices shown, such as Figure 17 As shown, the image processing device 10 includes an image acquisition module 11, an offset determination module 12, a sampling determination module 13, and a pixel determination module 14.

[0221] The image acquisition module 11 is used to acquire the image frame to be processed and the target permutation map corresponding to the image frame; the color channel of each first pixel in the target permutation map stores the displacement offset.

[0222] The offset determination module 12 is used to determine the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channel of each first pixel.

[0223] The sampling determination module 13 is used to determine the sampling pixel point in the image frame corresponding to each second pixel point based on the pixel displacement value of each second pixel point.

[0224] The pixel determination module 14 is used to replace the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel.

[0225] In an optional implementation, the image acquisition module 11 is used for:

[0226] Obtain the initial permutation map.

[0227] The aspect ratio of the image frame is obtained based on its resolution.

[0228] Based on the aspect ratio of the image frame, the initial permutation map is cropped to obtain the target permutation map.

[0229] In an optional implementation, the image acquisition module 11 is used for:

[0230] Based on the frame number of the image frame, an initial permutation map is determined in the preset permutation map sequence frame that matches the frame number of the image frame. The preset permutation map sequence frame includes multiple permutation maps, each with a frame number, and the displacement offset stored in the color channel of the first pixel in each permutation map is different.

[0231] In an optional implementation, the offset determination module 12 is used for:

[0232] Based on the preset offset threshold and scaling factor, the offset coefficient of each second pixel in the image frame is obtained.

[0233] For each second pixel in the image frame, based on the position information of the second pixel, the target first pixel whose position information matches the position information of the second pixel is obtained in the target permutation map.

[0234] The pixel displacement value of the second pixel is obtained based on the displacement offset and offset coefficient in the color channel of the first pixel.

[0235] In an optional embodiment, the image processing apparatus 10 may further include a color offset module, which, after acquiring the image frame to be processed, is used to:

[0236] Obtain the color information of each second pixel in each color channel and the color offset of each color channel.

[0237] For each color channel, based on the color offset of the color channel, determine the color offset pixel corresponding to each second pixel in the image frame, and replace the color information of each second pixel in the color channel with the color information of the color offset pixel corresponding to each second pixel in the color channel.

[0238] In an optional embodiment, the image processing apparatus 10 further includes a screen splitting module, which, after replacing the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel, is used to:

[0239] For each second pixel in the image frame, if the position information of the second pixel in the first direction is less than the first preset value, then the target pixel corresponding to the second pixel is determined according to the sum of the preset displacement value and the position information, and the pixel information of the second pixel is replaced with the pixel information of the target pixel.

[0240] If the position information of the second pixel in the first direction is greater than the second preset value, then the target pixel corresponding to the second pixel is determined according to the difference between the position information and the displacement value, and the pixel information of the second pixel is replaced with the pixel information of the target pixel. The second preset value is greater than the first preset value.

[0241] In an optional implementation, the split-screen module is used for:

[0242] Based on the frame number of the image frame, a target jitter value whose ordinal number matches the frame number is determined in a preset jitter value sequence; the jitter value sequence includes multiple jitter values, each jitter value having a unique ordinal number.

[0243] The target pixel corresponding to the second pixel is obtained by summing the displacement value, the target jitter value, and the position information.

[0244] Based on the difference between the position information and the displacement value, the target pixel corresponding to the second pixel is determined, including:

[0245] The target pixel corresponding to the second pixel is obtained based on the difference between the position information, displacement value, and target jitter value.

[0246] In an optional embodiment, the image processing apparatus 10 may further include a blurring module, which, after replacing the pixel information of each second pixel with the pixel information of the sampled pixel corresponding to each second pixel, is used to:

[0247] For each second pixel in the image frame, the second pixel is sampled in the second direction according to the preset blur radius to obtain the first sampled pixel.

[0248] Based on the blur radius, each of the first sampled pixels is sampled in the second direction to obtain the second sampled pixels.

[0249] The target pixel information of the second pixel is obtained by averaging the pixel information of each first sampled pixel and each second sampled pixel.

[0250] In an optional embodiment, the image processing apparatus 10 may further include an luminescence noise module, which, after replacing the pixel information of each second pixel with the pixel information of the sampled pixel corresponding to each second pixel, is used to:

[0251] Based on the frame number of the image frame, a target luminescent layer in the preset luminescent foreground sequence frame whose frame number matches that of the image frame is determined; the preset permutation map sequence frame includes multiple luminescent layers, each with a frame number.

[0252] Add the target glow layer to the image frame to obtain the image frame after adding the target glow layer.

[0253] Add a preset noise layer to the image frame after adding the target glow layer.

[0254] The image processing apparatus provided in this embodiment acquires the image frame to be processed and the target permutation map corresponding to the image frame. Based on the displacement offset stored in the color channel of each first pixel in the target permutation map, it determines the pixel displacement value of each second pixel in the image frame. Then, based on the pixel displacement value of each second pixel, it determines the sampling pixel in the image frame corresponding to each second pixel. After that, it replaces the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel. In this way, a fault effect with displacement can be achieved, enriching the visual experience and improving user stickiness. Furthermore, by storing the displacement offset in the color channel of the pixel in the permutation map and obtaining the pixel displacement value of each pixel in the image frame based on the displacement offset, the operation is simple and the response speed is fast, further improving user stickiness.

[0255] The image processing device provided in this embodiment can achieve a film scrolling effect with displacement faults by performing position offset, color offset, split-screen jitter blur processing, adding a glowing layer and a noise layer on each image frame, thereby enriching the special effects and improving user engagement.

[0256] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the image processing device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0257] Based on the above, this embodiment provides a readable storage medium storing a computer program, which, when executed by a processor, implements the image processing method of any of the foregoing embodiments.

[0258] The readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0259] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the readable storage medium described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0260] The foregoing has provided a detailed description of an image processing method, apparatus, electronic device, and readable storage medium provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of the present invention. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image frame to be processed and the target permutation map corresponding to the image frame; the color channel of each first pixel in the target permutation map stores the displacement offset; The pixel displacement value of each second pixel in the image frame is determined based on the displacement offset in the color channel of each first pixel. Based on the pixel displacement value of each second pixel, the sampling pixel point corresponding to each second pixel point in the image frame is determined; Replace the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel; The process of obtaining the target permutation map corresponding to the image frame includes: Based on the frame number of the image frame, an initial permutation map is determined in the preset permutation map sequence frame that matches the frame number of the image frame; the preset permutation map sequence frame includes multiple permutation maps, each permutation map has a frame number, and the displacement offset stored in the color channel of the first pixel in each permutation map is different; The aspect ratio of the image frame is obtained based on its resolution. Based on the aspect ratio of the image frame, the initial permutation map is cropped to obtain the target permutation map.

2. The image processing method according to claim 1, characterized in that, Determining the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channel of each first pixel includes: The offset coefficient of each second pixel in the image frame is obtained based on the preset offset threshold and scaling factor. For each second pixel in the image frame, based on the position information of the second pixel, a target first pixel whose position information matches the position information of the second pixel is obtained in the target permutation map; The pixel displacement value of the second pixel is obtained based on the displacement offset in the color channel of the first target pixel and the offset coefficient.

3. The image processing method according to claim 1, characterized in that, After acquiring the image frame to be processed, the method further includes: Obtain the color information of each second pixel in each color channel and the color offset of each color channel; For each color channel, based on the color offset of the color channel, determine the color offset pixel point in the image frame corresponding to each second pixel point, and replace the color information of each second pixel point in the color channel with the color information of the color offset pixel point corresponding to each second pixel point in the color channel.

4. The image processing method according to claim 1, characterized in that, After replacing the pixel information of each second pixel with the pixel information of the sampled pixel corresponding to each second pixel, the method further includes: For each second pixel in the image frame, if the position information of the second pixel in the first direction is less than a first preset value, then the target pixel corresponding to the second pixel is determined according to the sum of the preset displacement value and the position information, and the pixel information of the second pixel is replaced with the pixel information of the target pixel. If the position information of the second pixel in the first direction is greater than the second preset value, then the target pixel corresponding to the second pixel is determined according to the difference between the position information and the displacement value, and the pixel information of the second pixel is replaced with the pixel information of the target pixel, wherein the second preset value is greater than the first preset value.

5. The image processing method according to claim 4, characterized in that, The step of determining the target pixel corresponding to the second pixel based on the sum of the preset displacement value and the position information includes: Based on the frame number of the image frame, a target jitter value whose ordinal number matches the frame number is determined in a preset jitter value sequence; the jitter value sequence includes multiple jitter values, each jitter value having a unique ordinal number; The target pixel corresponding to the second pixel is obtained by summing the displacement value, the target jitter value, and the position information. The step of determining the target pixel corresponding to the second pixel based on the difference between the position information and the displacement value includes: The target pixel corresponding to the second pixel is obtained based on the difference between the position information, the displacement value, and the target jitter value.

6. The image processing method according to claim 1, characterized in that, After replacing the pixel information of each second pixel with the pixel information of the sampled pixel corresponding to each second pixel, the method further includes: For each second pixel in the image frame, the second pixel is sampled in the second direction according to a preset blur radius to obtain a first sampled pixel. Based on the blur radius, each of the first sampled pixels is sampled in the second direction to obtain the second sampled pixels; The target pixel information of the second pixel is obtained based on the average value of the pixel information of each of the first sampled pixels and each of the second sampled pixels.

7. The image processing method according to any one of claims 1-6, characterized in that, After replacing the pixel information of each second pixel with the pixel information of the sampled pixel corresponding to each second pixel, the method further includes: Based on the frame number of the image frame, a target luminous layer in a preset luminous foreground sequence frame whose frame number matches that of the image frame is determined; the preset luminous foreground sequence frame includes multiple luminous layers, each luminous layer having a frame number; Add the target luminescence layer to the image frame to obtain the image frame with the target luminescence layer added. Add a preset noise layer to the image frame after adding the target glow layer.

8. An image processing apparatus, characterized in that, The image processing device includes: An image acquisition module is used to acquire an image frame to be processed and, based on the frame number of the image frame, determine an initial permutation map in a preset permutation map sequence frame whose frame number matches the frame number of the image frame; obtain the aspect ratio of the image frame based on the resolution of the image frame; and crop the initial permutation map based on the aspect ratio of the image frame to obtain a target permutation map. The preset permutation map sequence frame includes multiple permutation maps, each permutation map has a frame number, and the displacement offset stored in the color channel of the first pixel in each permutation map is different. The first pixel in the target permutation map stores the displacement offset in its color channel. The offset determination module is used to determine the pixel displacement value of each second pixel in the image frame based on the displacement offset in the color channel of each first pixel. The sampling determination module is used to determine the sampling pixel point in the image frame corresponding to each of the second pixel points based on the pixel displacement value of each of the second pixel points; The pixel determination module is used to replace the pixel information of each second pixel with the pixel information of the sampling pixel corresponding to each second pixel.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium includes a computer program, which, when executed, controls the electronic device containing the readable storage medium to perform the image processing method according to any one of claims 1 to 7.

Images