Image processing method, device, electronic device and computer readable storage medium

Abstract

The present disclosure provides an image processing method, device, electronic device and computer-readable storage medium, the method comprising: obtaining a first image to be mounted and a first texture image, both of which are N-gons, where N is a positive integer greater than 2; segmenting the first image to be mounted to obtain a first n-gon of the first image to be mounted, where n is a positive integer greater than 2; segmenting the first texture image to obtain a second n-gon of the first texture image, where the first n-gon and the second n-gon have a first target mapping relationship; mapping each second n-gon to the corresponding first n-gon according to the first target mapping relationship to complete the texture mapping process of the first image to be mounted. The technical solution provided by the present disclosure can simplify the image rendering process in an augmented reality scenario and save server resources.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] Texture mapping is a process of mapping texture pixels in texture space to pixels in screen space. That is, a process of changing the appearance of a certain part of an image using an image, function or other data source. Currently, texture mapping is mainly performed on images through Open GL (OpenGraphics Library, a professional graphics programming interface with cross-programming language and cross-platform programming interface specifications) so that texture images can be mapped to screen spaces of different resolutions or sizes.

[0003] With the popularization of texture mapping technology, more and more software programs have added texture mapping technology in augmented reality scenarios, such as the construction of character frames in a certain game. Generally speaking, when the software is running, texture mapping technology will greatly occupy the computing resources of the mobile device CPU, resulting in frame dragging and frame drops when rendering the texture-mapped image to the screen.

[0004] Therefore, it is extremely meaningful to find a texture mapping method that occupies less CPU computing resources in augmented reality scenarios.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The embodiments of the present disclosure provide an image processing method and device, an electronic device, and a computer-readable storage medium, which can quickly and conveniently map a first texture image to a first image to be patched in an augmented reality scene.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] The disclosed embodiment proposes an image processing method, which includes: obtaining a first image to be patched and a first texture image, wherein the first image to be patched and the first texture image are both N-gons, where N is a positive integer greater than 2; performing segmentation processing on the first image to be patched to obtain a first n-gon of the first image to be patched, where n is a positive integer greater than 2; performing segmentation processing on the first texture image to obtain a second n-gon of the first texture image, wherein the first n-gon and the second n-gon have a first target mapping relationship; mapping each second n-gon to the corresponding first n-gon according to the first target mapping relationship, so as to perform texture mapping processing on the first image to be patched.

[0009] In some embodiments, the first n-gon is a first triangle, and the first image to be patched includes a target vertex; wherein, segmenting the first image to be patched to obtain the first n-gon of the first image to be patched includes: dividing each side of the first image to be patched into m equal parts to obtain m equally divided points on each side, where m is a positive integer greater than 1; and connecting the m equally divided points on each side of the first image to be patched with the target vertex to obtain the first n-gon.

[0010] In some embodiments, the first n-gon is a first triangle, and the first image to be pasted is a quadrilateral, and the quadrilateral includes a target opposite side; wherein, the first image to be pasted is segmented to obtain the first n-gon of the first image to be pasted, including: dividing the target opposite side of the first image to be pasted into m equal parts, obtaining m equal-divided points on the target opposite side, where m is a positive integer greater than 1; correspondingly connecting the m equal-divided points on the target opposite side of the first image to be pasted to obtain multiple target quadrilaterals; and connecting the diagonals of the multiple target quadrilaterals to obtain the first triangle.

[0011] In some embodiments, obtaining a first image to be pasted includes: obtaining an image to be processed including a target object; determining a standard image that matches the target object, the standard image including vertex feature points; determining target feature points in the image to be processed that match the vertex feature points of the standard image; connecting the target feature points in sequence to determine an image of the target object in the image to be processed; and determining the first image to be pasted based on the image of the target object.

[0012] In some embodiments, determining a standard image that matches the target object, the standard image including vertex feature points, includes: performing feature extraction processing on the image to be processed to obtain feature points of the image to be processed; and determining a standard image that matches the image to be processed based on the feature points of the image to be processed.

[0013] In some embodiments, obtaining the first image to be mounted further includes: obtaining an image to be processed including a target object; determining an image of the target object in the image to be processed by using an image recognition technology;

[0014] The first image to be mounted is determined according to the image of the target object.

[0015] In some embodiments, determining the first image to be mounted according to the image of the target object includes: adding a background to the image of the target object to generate the first image to be mounted, the target object being at a preset position of the first image to be mounted.

[0016] In some embodiments, the image processing method further includes: rendering the mapped first n-gon through a target rendering framework to display it on a target screen.

[0017] In some embodiments, the image processing method also includes: obtaining a second texture image, the second texture image and the first texture image both come from a target texture video; performing segmentation processing on the second texture image to obtain a third n-gon of the second texture image, the first n-gon and the third n-gon having a second target mapping relationship; mapping each third n-gon to a corresponding first n-gon according to the second target mapping relationship to perform video texture mapping processing on the first image to be patched.

[0018] In some embodiments, the image processing method also includes: obtaining a second image to be mounted, the first image to be mounted and the second image to be mounted are both acquired by a target device, and the second image to be mounted is the next frame image of the first image to be mounted; obtaining a second texture image, the second texture image and the first texture image are both from a target texture video, and the second texture image is the next frame image of the first texture image; mapping the second texture image to the second image to be mounted according to the above-mentioned image processing method; rendering the mapped first image to be mounted and the second image to be mounted in sequence to the target screen for display through a target rendering framework, so as to perform video texture mapping processing on the dynamic image acquired by the target device.

[0019] In some embodiments, obtaining a second image to be pasted includes: performing image acquisition on the target object through the target device to obtain the image to be processed at the current moment; obtaining the image to be processed at the previous moment acquired by the target device, the image to be processed at the previous moment including vertex feature point information of the target object; processing the vertex feature point information of the target object at the previous moment to obtain the vertex feature point information of the target object in the image to be processed at the current moment; determining the image of the target object at the current moment in the image to be processed at the current moment according to the vertex feature point information of the target object at the current moment; and determining the second image to be pasted according to the image of the target object at the current moment.

[0020] The embodiment of the present disclosure provides an image processing device, which includes: an image acquisition module, a first segmentation module, a second segmentation module and a mapping module.

[0021] Among them, the image acquisition module can be configured to acquire a first image to be patched and a first texture image, wherein the first image to be patched and the first texture image are both N-gons, where N is a positive integer greater than 2; the first segmentation module can be configured to perform segmentation processing on the first image to be patched to obtain a first n-gon of the first image to be patched, where n is a positive integer greater than 2; the second segmentation module can be configured to perform segmentation processing on the first texture image to obtain a second n-gon of the first texture image, wherein the first n-gon and the second n-gon have a first target mapping relationship; the mapping module can be configured to map each second n-gon to the corresponding first n-gon according to the first target mapping relationship, so as to perform texture mapping processing on the first image to be patched.

[0022] An embodiment of the present disclosure provides an electronic device, which includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned image processing methods.

[0023] The embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method as described in any one of the above items is implemented.

[0024] The image processing method, device, electronic device and computer-readable storage medium provided by the embodiment of the present disclosure obtain the first n-gon and the second n-gon with a mapping relationship by segmenting the first image to be patched and the first texture image; then the second n-gon is directly mapped to the first n-gon according to the mapping relationship to complete the mapping operation of patching the first texture image to the first image to be patched. This method, on the one hand, does not need to obtain the pose matrix between the first image to be patched and the first texture image, and then does not need to map the first texture image to the coordinate system where the first image to be patched is located, which greatly saves CPU computing resources and avoids the problem of dragging or dropping frames; on the other hand, this method can adjust the number of the first n-gon according to the performance of the device (including the device CPU and GPU). When the performance of the device is good, the number of the first n-gon can be large to reduce the deformation caused by patching. When the performance of the device is poor, the number of the first n-gon can be small to avoid the problems of dropping or dragging frames caused by the device running jam. Since the number of the first n-gons (or the second n-gons) is adjustable, the method can be adapted to various devices without algorithm optimization, so that the display effect of the mapped image on various devices is consistent.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. The drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work.

[0027] Figure 1 A schematic diagram showing an exemplary system architecture of an image processing method or an image processing device applied to an embodiment of the present disclosure.

[0028] Figure 2 The diagram is a schematic structural diagram of a computer system applied to an image processing apparatus according to an exemplary embodiment.

[0029] Figure 3 It is a schematic diagram of a P4P algorithm solution according to the related technology.

[0030] Figure 4 The invention discloses an image processing system according to the related art.

[0031] Figure 5 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0032] Figure 6 The figure is a schematic diagram of image segmentation according to an exemplary embodiment.

[0033] Figure 7 The figure is a schematic diagram of image segmentation according to an exemplary embodiment.

[0034] Figure 8 The figure is a schematic diagram of image deformation according to an exemplary embodiment.

[0035] Fig. 9 The figure is a schematic diagram of image segmentation according to an exemplary embodiment.

[0036] Fig.10 yes Figure 5 Flowchart of step S01 in an exemplary embodiment.

[0037] Fig.11 The figure is a schematic diagram showing how to determine a target object in an image to be processed according to an exemplary embodiment.

[0038] Fig.12 The figure is a schematic diagram showing how to determine a target object in an image to be processed according to an exemplary embodiment.

[0039] Fig.13 is a shape of an image of a target object separated from an image to be processed according to an exemplary embodiment.

[0040] Fig.14 yes Fig.10 Flowchart of step S012 in an exemplary embodiment.

[0041] Fig.15 yes Fig.10 Flowchart of step S014 in an exemplary embodiment.

[0042] Fig.15A The figure is a schematic diagram of an image after rendering according to an exemplary embodiment.

[0043] Fig.16 yes Figure 5 Flowchart of step S01 in an exemplary embodiment.

[0044] Fig.17 yes Figure 5 Flowchart of step S02 in an exemplary embodiment.

[0045] Fig.18 The figure is a schematic diagram of image segmentation according to an exemplary embodiment.

[0046] Fig.19 yes Figure 5Flowchart of step S02 in an exemplary embodiment.

[0047] Fig. 20 The invention discloses an image segmentation method according to an exemplary embodiment.

[0048] Fig.21 The invention relates to an image processing method implemented according to an exemplary embodiment.

[0049] Fig. 22 The invention discloses an image processing method according to an exemplary embodiment.

[0050] Fig.23 yes Fig. 22 Step S08 in the flowchart is shown in an exemplary embodiment.

[0051] Fig.24 The invention discloses an image processing system according to an exemplary embodiment.

[0052] Fig.25 The invention discloses an image processing system according to an exemplary embodiment.

[0053] Fig.26 is a block diagram of an image processing apparatus according to an exemplary embodiment. DETAILED DESCRIPTION

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0055] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0056] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0057] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0058] In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of one or more elements / components / etc.; the terms "comprising", "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0059] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0060] Figure 1 A schematic diagram showing an exemplary system architecture that can be applied to an image processing method or an image processing device according to an embodiment of the present disclosure.

[0061] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0062] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, virtual reality devices, smart homes, etc.

[0063] The server 105 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received request data, and feed back the processing results to the terminal device.

[0064] The server 105 may, for example, obtain a first image to be patched and a first texture image, wherein both the first image to be patched and the first texture image are N-gons, where N is a positive integer greater than 2; the server 105 may, for example, perform segmentation processing on the first image to be patched to obtain a first n-gon of the first image to be patched, where n is a positive integer greater than 2; the server 105 may, for example, perform segmentation processing on the first texture image to obtain a second n-gon of the first texture image, where the first n-gon and the second n-gon have a first target mapping relationship; the server 105 may, for example, map each second n-gon to the corresponding first n-gon according to the first target mapping relationship to perform texture mapping processing on the first image to be patched.

[0065] It should be understood that Figure 1 The number of terminal devices, networks and servers in the figure is only for illustration. The server 105 may be a physical server or may be composed of multiple servers. According to actual needs, it may have any number of terminal devices, networks and servers.

[0066] Reference below Figure 2 , which shows a schematic diagram of the structure of a computer system 200 of a terminal device suitable for implementing an embodiment of the present application. Figure 2 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0067] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the system 200 are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0068] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0069] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 209, and / or installed from a removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the above-mentioned functions defined in the system of the present application are executed.

[0070] It should be noted that the computer-readable storage medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0071] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0072] The modules and / or units and / or sub-units involved in the embodiments of the present application may be implemented in software or in hardware. The modules and / or units and / or sub-units described may also be arranged in a processor, for example, may be described as: a processor includes a sending unit, an acquiring unit, a determining unit, and a first processing unit. The names of these modules and / or units and / or sub-units do not, in some cases, constitute limitations on the modules and / or units and / or sub-units themselves.

[0073] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by a device, the device can realize functions including: obtaining a first image to be patched and a first texture image, wherein the first image to be patched and the first texture image are both N-gons, where N is a positive integer greater than 2; segmenting the first image to be patched to obtain a first n-gon of the first image to be patched, where n is a positive integer greater than 2; segmenting the first texture image to obtain a second n-gon of the first texture image, wherein the first n-gon and the second n-gon have a first target mapping relationship; mapping each second n-gon to the corresponding first n-gon according to the first target mapping relationship to perform texture mapping on the first image to be patched.

[0074] With the popularization of computer technology, the application of image mapping technology is becoming more and more extensive. For example, it can be used in image beautification (for example, pasting glasses on a face image), in image introduction (when scanning a target image, related images can be displayed on the target image or around the target image, etc.), and in the construction of 3D characters in games.

[0075] In the related art, PNP (Perspective-N-Point) technology is usually used to map the texture image in the texture space to the image to be pasted, and then the rendering framework renders the image to be pasted to the screen for display.

[0076] Generally speaking, PNP technology can calculate the projection relationship between the coordinate system of the texture space and the coordinate system of the image to be pasted through N feature points in the coordinate system of the texture space and N corresponding feature points in the coordinate system of the image to be pasted, thereby obtaining the camera pose or object pose (that is, obtaining the MVP (Model-View-Projection) matrix).

[0077] Currently, OpenCV (a cross-platform computer vision library), a commonly used image processing library in computer vision, supports at least three methods for solving the PNP problem, namely P3P (Perspective-3-Point), EPNP (Efficient Perspective-N-Point) and ITERATIVE (a default PNP solution method).

[0078] The following will take the P4P (Perspective-4-Point, 4-point perspective) method as an example to explain the specific steps of the image mapping method in the related art.

[0079] The P4P method first solves four possible matrix transformation relationships through the three feature points in the coordinate system of the texture image and the corresponding three feature fixed points in the coordinate system of the image to be pasted (the three feature points in the coordinate system of the texture image and the three feature points in the coordinate system of the image to be pasted should have a one-to-one correspondence), and then reprojects the fourth feature point in the texture image to the coordinate system of the texture image through the above four matrix transformation relationships, and determines the final matrix transformation relationship (that is, the MVP matrix, that is, the pose matrix) by minimizing the reprojection error technology.

[0080] The solution process of P4P technology will be explained below.

[0081] Figure 3 It is a P4P solution schematic diagram shown according to the relevant technology.

[0082] Assume that P is the camera corresponding to the image to be patched, A, B, C, and D are the four feature points in the target texture image (assuming that the target texture image is in the world coordinate system), and a, b, c, and d are the four feature points in the image to be patched (assuming that the image to be patched is in the pixel coordinate system).

[0083] like Figure 3 As shown, four groups of approximate camera poses are first solved based on A, B, C and a, b, c.

[0084] In this embodiment, the camera intrinsic matrix and distortion parameters are known. Through the camera intrinsic matrix and distortion parameters and the coordinates of a, b, and c, the coordinates of A, B, and C at point P, that is, the camera space, can be obtained by using the similarity of the triangle space. According to the cosine theorem:

[0085] PA 2 +PB 2 -2PA·PBcos<a,b> = AB 2 (1)

[0086] PA 2 +PC2 -2PA·PCcos<a,c> = AC 2 (2)

[0087] PB 2 +PC 2 -2PB·PBcos<b,c> = BC 2 (3)

[0088] Divide by PC 2 Eliminate variables and use the following formula for substitution:

[0089]

[0090] After substitution, formulas (1) to (3) can be rewritten as formulas (4) to (6):

[0091] x 2 +y 2 -2x·ycos<a,b> = u (4)

[0092] x 2 +1-2xcos<a,c> = wu (5)

[0093] y 2 +1-2ycos<b,c> = vu (6)

[0094] Where w, v, cos<a,b> , cos<a,c> , cos<b,c> As is known, by solving formula (4) to formula (6), four MVP matrices can be finally obtained.

[0095] By combining the feature point D with the above four MVP matrices, the four mapping points of the feature point D can be found in the coordinate system where abc is located, and then the reprojection error between these four mappings and point d is calculated, and the MVP matrix corresponding to the minimum error is determined to be the appropriate solution.

[0096] After obtaining the final MVP matrix through P4P technology, all points in the coordinate system of ABCD can be mapped to the coordinate system of abcd, thereby completing the image mapping process.

[0097] Figure 4 The invention discloses an image processing system according to the related art. Figure 4 The image processing method shown may adopt the above mapping method. Specifically, it may include the following steps:

[0098] The target user 401 scans the target object through the target device 402 to obtain the image to be processed (it can be understood that the image to be processed includes not only the target object but also the background information); the target device 402 uploads the image to be processed to the server 403 so that the server 403 determines the target object image from the image to be processed; the server 403 returns the standard image, texture image and four target feature point information in the standard image of the target object, and the corresponding four target feature point information in the image to be pasted to the target device 402 (wherein the standard image and the texture image should be in the same coordinate system); the target device 402 can process the target feature point information in the standard image and the image to be pasted through the P4P technology to obtain the MVP matrix between the coordinate system of the standard image and the coordinate system of the image to be pasted; the target device 402 maps the texture image to the image to be pasted according to the MVP matrix between the coordinate system of the standard image and the coordinate system of the image to be pasted, and renders the image to be pasted onto the screen of the target device 402 through the target rendering framework (such as OpenGL).

[0099] In the above embodiment, the existing image processing system needs to use P4P technology to map the texture image to the image to be pasted, and the P4P algorithm solution requires a large amount of calculation, which occupies the CPU computing resources of the device, resulting in a weakening of the CPU rendering capability. In the related art, if synchronous calculation rendering is used, it will cause frame drops, and when the frame rate is lower than 30 frames, the user will see obvious freezes. If asynchronous calculation rendering is used, the video texture and the camera texture cannot be accurately matched, and the user side will see the phenomenon of dragging frames.

[0100] The disclosed embodiment provides an image mapping method that occupies less algorithm CPU computing resources to improve image rendering capabilities.

[0101] Figure 5 is a flowchart of an image processing method according to an exemplary embodiment. The method provided by the embodiment of the present disclosure can be processed by any electronic device with computing and processing capabilities, such as the above Figure 1 The server 105 and / or terminal devices 102 and 103 in the embodiment are described by taking the server 105 as the execution subject in the following embodiment, but the present disclosure is not limited to this.

[0102] Reference Figure 5 The image processing method provided by the embodiment of the present disclosure may include the following steps.

[0103] In step S01, a first image to be mounted and a first texture image are obtained, wherein the first image to be mounted and the first texture image are both N-gons, where N is a positive integer greater than 2.

[0104] When visiting a museum or art gallery, a user may use a target device (such as a mobile phone) to scan a corresponding painting and hope to see other works or text introductions related to the painting at the same time. In this case, it may be necessary to perform image mapping or video mapping on the image scanned by the target device to map other related works or introductions onto the scanned painting.

[0105] In some embodiments, the first image to be mounted may be any photo that needs to be mapped, and the first texture image may be an image that needs to be pasted onto the image to be mounted.

[0106] It should be noted that, in order to better paste the first texture image onto the first image to be pasted, the first texture image and the first image to be pasted may include the same number of edges. For example, if the first image to be pasted is a quadrilateral, then the first texture image should also be a quadrilateral.

[0107] It can be understood that in order to ensure the mapping effect, it can be ensured that the number of sides of the first texture image and the first image to be mounted are consistent and the ratio of the side lengths is approximately consistent (for example, the first image to be mounted is a rectangle, and the ratio of the left and top is 1:2, and the first texture image is also a rectangle, and the ratio of the left and top is also 1:2).

[0108] In step S02, the first image to be mounted is segmented to obtain a first n-gon of the first image to be mounted, where n is a positive integer greater than 2.

[0109] In some embodiments, the first n-gon may be a triangle, or may be a quadrilateral, a pentagon or other shapes, which is not limited in the present disclosure.

[0110] However, since the basic rendering element of an existing commonly used image rendering framework (such as OpenGL) is a triangle, the present disclosure will be described by taking the first n-gon as a triangle as an example.

[0111] In some embodiments, if the first image to be mounted is Figure 6 The quadrilateral shown in FIG. 1 can be connected at its diagonal angles to split the quadrilateral into two triangles for mapping operations. If the first image to be pasted is as follows Figure 7 For other polygons (such as pentagons) shown, a vertex of the pentagon can be connected to other vertices to split the pentagon into three triangles for mapping operations. Obviously, there are many ways to split the image to be pasted into triangles, and the present disclosure does not limit this.

[0112] The basic idea of ​​the embodiment of the present disclosure is to segment the first image to be pasted and the first texture image using the same segmentation method (for example, connecting the diagonals of the quadrilaterals) to segment the first image to be pasted and the first texture image into multiple triangles, and then paste the triangles of the first texture image onto the corresponding triangles of the first image to be pasted.

[0113] When the triangles of the first texture image are mapped onto the triangles of the first image to be patched, there will be a certain amount of splicing between the triangles of the first texture image. However, since the proportions between the shapes of the first texture image and the first image to be patched may not be completely consistent, the proportions between the shapes of the triangles segmented from the first texture image and the first image to be patched may not be consistent. Therefore, when pasting the triangles of the first texture image onto the corresponding triangles on the first patch image, it is necessary to perform certain scaling and stretching on the triangles of the first texture image. However, once the triangles corresponding to the first texture image are stretched and scaled, the following problems will inevitably occur during splicing. Figure 8 The deformation shown (which is an obvious positioning problem, Figure 8 The triangles are rasterized). Or it can be understood that since the vertex triangles in the rendering framework are directional, the following will be generated Figure 8 The above deformation is shown at the diagonal stitching position, and this deformation is not obvious when the camera is initially scanning the work (because the two rendered triangles have the same scaling ratio, or the same direction of the lines).

[0114] In order to make the deformation less obvious, this embodiment divides the image to be pasted into a plurality of first n-gons (such as triangles) to evenly distribute the deformation to each small first n-gon.

[0115] For example, assuming that the first n-gon is a first triangle, and the first image to be mounted includes a target vertex, each side of the first image to be mounted can be divided into m equal parts to obtain m equally divided points on each side, where m is a positive integer greater than 1; then the m equally divided points on each side of the first image to be mounted are connected with the target vertex to obtain the first n-gon.

[0116] In addition, if the first n-gon is a first triangle, the first image to be pasted is a quadrilateral, and the quadrilateral includes the target opposite side. Fig. 9 As shown in the left figure, the target opposite sides of the first image to be mounted can also be divided into m equal parts to obtain m equal parts points on the target opposite sides, where m is a positive integer greater than 1; then the m equal parts points on the target opposite sides of the first image to be mounted are connected correspondingly to obtain multiple target quadrilaterals; finally, the diagonals of the multiple target quadrilaterals are connected to obtain the first triangle.

[0117] It is found from actual measurements that the more the number of first n-gons is, the more vertices are required to be rendered, and the higher the performance requirements for the device are. Currently in practical applications, the number of first n-gons can be 12 to 30.

[0118] In step S03, segmentation processing is performed on the first texture image to obtain a second n-gon of the first texture image, and the first n-gon and the second n-gon have a first target mapping relationship.

[0119] In some embodiments, the same segmentation method can be used to segment the first texture image to obtain a second n-gon corresponding to the first segmented image. It should be understood that the number of sides of the second n-gon is the same as the number of sides of the first n-gon. If the first n-gon is a 4-gon, then the second n-gon should also be a 4-gon, and there should also be a first target mapping relationship between the first n-gon and the second n-gon. The so-called first mapping relationship may refer to the corresponding relationship between the first n-gon and the second n-gon. Fig. 9 As shown, Fig. 9 (a) is the first triangle obtained after segmenting the patch image. Fig. 9 (b) is the second target triangle obtained after segmenting the first texture image. It is obvious that the first triangle S1 and the second triangle S2 have a corresponding mapping relationship. The second triangle S2 can be mapped to the first triangle S1 to realize the image mapping operation.

[0120] In step S04, each second n-gon is mapped to the corresponding first n-gon according to the first target mapping relationship, so as to perform texture mapping processing on the first image to be pasted.

[0121] In some embodiments, during mapping, the vertices of two triangles may be aligned, and then stretched and shrunk. Of course, the present disclosure does not limit the specific mapping method.

[0122] In some embodiments, after the mapping process of the image to be patched is completed, the first n-gon after mapping can be rendered by the target rendering framework to be displayed on the target screen. The target rendering framework can be an OpenGL rendering engine, and can also be used in other rendering engines such as Metal (a rendering application programming interface), which is not limited by the present disclosure.

[0123] The image processing method provided in this embodiment obtains a first n-gon and a second n-gon with a mapping relationship by segmenting the first image to be pasted and the first texture image; then the second n-gon is directly mapped to the first n-gon according to the mapping relationship to complete the mapping operation of pasting the first texture image to the first image to be pasted. This method, on the one hand, does not need to obtain the pose matrix between the first image to be pasted and the first texture image, and then does not need to map the first texture image to the coordinate system where the first image to be pasted is located, which greatly saves CPU computing resources and avoids the phenomenon of dragging or dropping frames during the rendering process; on the other hand, this method can adjust the number of the first n-gon according to the performance of the device. When the performance of the device is good, the number of the first n-gon can be larger to reduce the deformation caused by pasting. When the performance of the device is poor, the number of the first n-gon can be smaller, which further avoids the problems of dropping or dragging frames caused by the device running jam. Since the number of the first n-gons (or the second n-gons) is adjustable, the method can be adapted to various devices without algorithm optimization, so that the display effect of the mapped image on various devices is consistent.

[0124] Fig.10 yes Figure 5 Flowchart of step S01 in an exemplary embodiment. Fig.10 , the above step S01 may include the following steps.

[0125] In step S011, an image to be processed including a target object is acquired.

[0126] In some embodiments, the image to be processed may include not only information about the target object (any object or thing that can be captured) but also background information. For example, in an art gallery, if a user scans a target image, then the image including background information obtained by the target user scanning may be the image to be processed. Fig.11 (1) or Fig.12 The image shown in (1) may be the image to be processed.

[0127] In step S012, a standard graph matching the target object is determined, where the standard graph includes vertex feature points.

[0128] In some embodiments, the standard image may refer to an original image that only includes the target object, such as Fig.11 (2) or Fig.12 (2) as shown. Fig.11 (2) It can be Fig.11 The standard graph of the target object in (1), Fig.12 (2) It can be Fig.12 The standard graph of the target object in (1).

[0129] When making a mapping system for paintings in a museum, the corresponding standard map can be extracted and stored in advance for each painting, so that the image of the target object can be determined in the image to be processed obtained by the target user scanning according to the standard map.

[0130] Of course, if the target object and the background in the image to be processed are clearly separated, the grayscale image of the image to be processed can be directly used in combination with edge recognition technology to determine the image of the target object and the vertex feature point information of the image of the target object from the image to be processed.

[0131] In some embodiments, the standard graph may also be an N-gon, which includes multiple feature point information, wherein the multiple feature point information includes N vertex feature point information (for example, coordinate information of the N vertex feature points in the coordinate system of the standard graph).

[0132] In step S013, target feature points matching the vertex feature points of the standard image are determined in the image to be processed.

[0133] In some embodiments, features of the image to be processed may be extracted using image feature point extraction technology, and feature matching may be performed in a standard image library using image matching technology to determine a standard image that matches the image to be processed from multiple standard images in the standard library.

[0134] In some embodiments, after determining a standard image that matches the image to be processed, a target feature point is determined in the image to be processed based on the N vertex feature points of the standard image. The target feature point should be a vertex feature point of the target object in the image to be processed. Fig.11 As shown, Fig.11 (1) represents the image to be processed, which includes not only the target object but also the background information. Fig.11 (2) A standard image representing the image to be processed. Fig.11 As shown, the target feature point abcd matching the feature point ABCD (vertices of the target object) can be determined in the image to be processed by feature matching technology. Fig.12 As shown, Fig.12 (1) represents the image to be processed obtained by the image acquisition device from a non-frontal angle to the target object, Fig.12 (2) A standard image representing the target object, through feature matching technology, can determine the target feature point abcd matching the feature point ABCD in the image to be processed. It can be understood that here only the standard image is a quadrilateral as an example for description, but the present disclosure is not limited to this.

[0135] In step S014, the target feature points are connected in sequence to determine the image of the target object in the image to be processed.

[0136] In some embodiments, Fig.11 In (1), the target feature points abcd are Fig.11 The sequential connection of the vertex feature points ABCD in (2) can separate the image of the target object from the image to be processed.

[0137] It should be understood that the number of edges of the image of the target object segmented from the first image to be processed should be consistent with that of the standard image.

[0138] Fig.13 is the shape of the image of the target object separated from the image to be processed according to an exemplary embodiment. Fig.13 As shown, Fig.13 The upper part shows the images to be processed obtained by capturing the target object from various angles. Fig.13 The lower part shows the shapes of the images of the target objects separated from each image to be processed. Obviously, different shapes can be obtained by capturing the target object from different angles.

[0139] Step S015: determining the first image to be mounted according to the image of the target object.

[0140] In some embodiments, the image of the target object can be the first image to be mounted, or a part of the first image to be mounted (for example, the first image to be mounted includes not only the image of the target object, but also some set background), and the first image to be mounted can also be a part of the image of the target object (for example, the target object is a face, and the first image to be mounted should be the eye part of the face image). It should be understood that the present disclosure does not limit the content of the first image to be mounted, and the technician can obtain the first image to be mounted according to his own needs.

[0141] It should be noted that if the first image to be mounted is the image of the target object determined according to the standard image, the number of edges of the standard image should also be N; if the first image to be mounted is not the image of the target object, the present disclosure does not limit the number of edges of the standard image.

[0142] In this embodiment, the standard image of the target object is determined by image matching technology, and the image of the target object is determined from the image to be processed by the vertex feature points of the standard image. Compared with the image segmentation method, this method can accurately segment the image of the target object from the image to be processed with an unclear background.

[0143] Fig.14 yes Fig.10 Flowchart of step S012 in an exemplary embodiment. Fig.14 , the above step S012 may include the following steps.

[0144] In step S0121, feature extraction is performed on the image to be processed to obtain feature points of the image to be processed.

[0145] In some embodiments, feature extraction algorithms may be used to extract features from the image to be processed, such as SIFT (Scale-invariant features transform), SURF (Speeded Up Robust Features), etc. Some main features can be extracted from the image to be processed through the above image feature extraction algorithms.

[0146] In step S0122, a standard image matching the image to be processed is determined according to the feature points of the image to be processed.

[0147] In some embodiments, a standard image matching the image to be processed can be determined in a standard image library by using the features of the image to be processed. It is understood that the standard images in the standard image library all include multiple feature points, and the feature points of the standard image can be matched with the feature points of the image to be processed by an image matching algorithm such as SIFT to determine the standard image of the image to be processed.

[0148] This embodiment can accurately determine the standard image of the target object in the standard image library through image matching technology.

[0149] Fig.15 yes Fig.10 Flowchart of step S014 in an exemplary embodiment. Fig.15 , the above step S014 may include the following steps.

[0150] In step S0141, an image to be processed including a target object is acquired.

[0151] In step S0142, a standard graph matching the target object is determined, where the standard graph includes vertex feature points.

[0152] In step S0143, the feature points that match the multiple vertices of the standard graph are connected in sequence to determine the target object in the image to be processed.

[0153] In some embodiments, the image of the target object may be the first image to be mounted, or may be a part of the first image to be mounted (for example, the first image to be mounted includes not only the image of the target object, but also some set background), and the first image to be mounted may also be a part of the image of the target object (for example, the target object is a face, and the first image to be mounted should be the eye part of the face image). It should be understood that the present disclosure does not limit the content of the first image to be mounted, and technicians can obtain the first image to be mounted according to their own needs.

[0154] In step S0144, a background is added to the image of the target object to generate the first image to be mounted, and the target object is located at a preset position of the first image to be mounted.

[0155] In some embodiments, the first image to be pasted may be an image including both the target object and the background image. For example, if you want to paste an image next to the target object image in a scene, you need to add a background around the target object. It is understandable that the present disclosure does not limit the color, size, type, transparency, etc. of the background.

[0156] In some embodiments, the above image processing method can be used to render Fig.15A Image shown. Fig.15A The figure is a schematic diagram of an image after rendering according to an exemplary embodiment.

[0157] like Fig.15A As shown, if the mapping position of texture image 1503 is not limited to the target object image 1501, it is necessary to add background 1502 to the target object 1501. Of course, it is generally necessary to add a transparent background to texture image 1503 so that texture image 1503 can be pasted to a suitable position.

[0158] In some embodiments, the preset position in this embodiment can be set according to actual needs. For example, the target object image can be set in the middle, left or right position of the first image to be mounted, etc., and the present disclosure does not limit this.

[0159] The technical solution provided in this embodiment can process the image of the target object to obtain the image to be mounted required by the technician.

[0160] Fig.16 yes Figure 5 Flowchart of step S01 in an exemplary embodiment. Fig.16 , the above step S01 may include the following steps.

[0161] In step S015, an image to be processed including the target object is acquired.

[0162] In step S016, an image of a target object is determined in the image to be processed by using image recognition technology to determine the first image to be mounted.

[0163] The technical solution provided in this embodiment determines the image of the target object from the image to be processed by image recognition technology (for example, using image segmentation technology to perform edge recognition processing on the grayscale image). Compared with determining the image of the target object in the image to be processed by annotating the image, it saves CPU computing resources and improves the image recognition speed.

[0164] Fig.17 yes Figure 5 Flowchart of step S02 in an exemplary embodiment.

[0165] In some embodiments, the first n-gon may be a triangle, the first image to be patched includes a target vertex, and the target vertex may be any vertex of the first image to be patched, which is not limited in the present disclosure.

[0166] refer to Fig.17 , the above step S02 may include the following steps.

[0167] In step S021, each side of the first image to be mounted is divided into m equal parts to obtain m equal division points on each side, where m is a positive integer greater than 1.

[0168] In some embodiments, the image to be pasted may be a triangle, a quadrilateral, a pentagon... This disclosure does not impose any limitation on this.

[0169] like Fig.18 If the image to be patched is a triangle, you can use Fig.18 (a) The method shown in Figure 1 divides each edge into m equal parts (where m = 1, 2, 3, etc.). If the image to be patched is a pentagon, the following method can be used: Fig.18 The method shown in (b) divides each side of the pentagon into m equal parts (here m=1, 2, 3, ...) It can be understood that the present disclosure does not limit the image segmentation method.

[0170] In step S022, m equally divided points on each side of the first image to be pasted are connected to the target vertex to obtain the first n-gon.

[0171] like Fig.18 As shown, after each side of the first image to be mounted is divided into m equal parts, the m equal-divided points on each side of the first image to be mounted can be connected with the target vertex to obtain the first n-gon.

[0172] In some embodiments, the target vertex may be any vertex of the first image to be tiled, which is not limited in the present disclosure.

[0173] It should be understood that any method that can divide the first image to be tiled into first n-gons is within the protection scope of the present disclosure.

[0174] It should be noted that the present disclosure requires that the same segmentation method be used for the first image to be patched and the first texture image, so the target vertices in the first image to be patched and the target vertices in the first texture image should also correspond.

[0175] The present embodiment provides an image segmentation method, which can ensure that the first image to be pasted is segmented into any number of first n-gons, and because the number of first n-gons is controllable, it is ensured that the method can optimize the image mapping technology at any time for hardware devices of different specifications (that is, determine the number of first n-gons for different hardware devices so that the rendering effect of the image on different devices is the same).

[0176] Fig.19 yes Figure 5 Flowchart of step S02 in an exemplary embodiment.

[0177] In some embodiments, the first n-gon is a first triangle, the first image to be patched is a quadrilateral, and the quadrilateral includes a target opposite side.

[0178] refer to Fig.19 , the above step S02 may include the following steps.

[0179] In step S023, the target opposite sides of the first image to be mounted are divided into m equal parts to obtain m equal-divided points on the target opposite sides, where m is a positive integer greater than 1.

[0180] like Fig. 20 As shown in (a), a pair of sides (such as AB and CD) of the first image ABCD to be patched can be divided into m equal parts to obtain Fig. 20 (b) The image shown.

[0181] In step S024, the m equally divided points on the opposite sides of the target of the first image to be pasted are correspondingly connected to obtain a plurality of target quadrilaterals.

[0182] like Fig. 20 As shown in (c), the m equally divided points on the opposite sides AB and CD can be connected correspondingly to obtain the corresponding target quadrilateral. It should be understood that the target quadrilateral generated in the process of segmenting the first image to be patched can be an approximate parallelogram, and the present disclosure does not limit this.

[0183] In step S025, the diagonals of the plurality of target quadrilaterals are connected to obtain the first triangle.

[0184] In some embodiments, one diagonal of the target parallelogram may be connected, or both diagonals may be connected. The technicians may make settings according to actual needs, and the present disclosure does not impose any restrictions on this.

[0185] The present embodiment provides an image segmentation method, which can not only segment a first image to be pasted (quadrilateral) into any number of first n-gons, but also because the number of first n-gons is controllable, ensures that the method can optimize the image mapping technology at any time for hardware devices of different specifications (that is, determine the number of first n-gons for different hardware devices so that the rendering effect of the image on different devices is the same).

[0186] Fig.21 is an image processing method implemented according to an exemplary embodiment. Fig.21 , the above-mentioned image processing method may include the following steps.

[0187] In some embodiments, not only the texture image can be mapped to the first image to be mounted, but also the texture video can be mapped to the first image to be mounted.

[0188] In step S05, a second texture image is obtained, where the second texture image and the first texture image both come from a target texture video.

[0189] In some embodiments, if the target texture video is to be mapped to the first image to be mounted, the target texture video may be split into frames of images according to time, and the frames of images may be sequentially mapped to the first image to be mounted according to time.

[0190] That is, after mapping the first target texture image to the first image to be patched, the next frame of image can be obtained from the target texture video (it can be assumed that the next frame of image is the second texture image). It can be understood that the next frame of image of the first target texture image can be directly obtained from the original target texture video, or the target texture video can be sampled (for example, sampled at a certain frame rate) to obtain the next frame of image, and the present disclosure does not limit this.

[0191] In step S06, the second texture image is segmented to obtain a third n-gon of the second texture image, and the first n-gon and the third n-gon have a second target mapping relationship.

[0192] In some embodiments, the second texture image may be segmented according to the segmentation method of the first image to be patched to obtain a third n-gon, wherein the third n-gon has the same number of sides as the first n-gon, and the third n-gon and the first n-gon have a second target mapping relationship.

[0193] In step S07, each third n-gon is mapped to the corresponding first n-gon according to the second target mapping relationship, so as to perform video texture mapping processing on the first image to be pasted.

[0194] It is understandable that after pasting the second texture image, the next frame image can be obtained in the target texture video to continue the pasting operation of the target texture video.

[0195] The technical solution provided in this embodiment can map the target texture video onto the image to be pasted, and the final visual effect obtained is that the target object in the image to be pasted seems to be moving.

[0196] Fig. 22 is an image processing method according to an exemplary embodiment. Fig. 22 , the above-mentioned image processing method may include the following steps.

[0197] In some embodiments, not only can the target texture video be mapped onto the first image to be mapped, but the target texture video can also be mapped onto the video to be mapped. For example, when a user scans a target object through a mobile device, the user can not only see the target object in real time through the mobile device, but also see the target texture video pasted on the target object image in real time.

[0198] refer to Fig. 22 , the above-mentioned image processing method may include the following steps.

[0199] In step S08, a second image to be mounted is obtained, wherein the first image to be mounted and the second image to be mounted are both obtained by scanning the target device, and the second image to be mounted is the next frame image of the first image to be mounted.

[0200] In some embodiments, when a user scans a target object through a mobile device, since the mobile device scans the target object according to a target frame rate, the picture viewed by the user through the mobile device can be considered as the video to be pasted.

[0201] In some embodiments, both the first image to be mounted and the second image to be mounted may be obtained by scanning the target object by a mobile device, wherein the first image to be mounted may be a previous frame image of the second image to be mounted.

[0202] In step S09, a second texture image is obtained, wherein the second texture image and the first texture image are both from a target texture video, and the second texture image is a next frame image of the first texture image.

[0203] In some embodiments, the target texture video can be sampled according to the target frame rate of the mobile device, so that the sampled frames can be pasted on the frames scanned by the mobile device in time. That is, assuming that the mobile device obtains 30 frames of images per second, there will be 30 frames of images per second after sampling the target texture video, and the 30 frames of the target texture video can be mapped to the 30 images scanned by the mobile device in time order.

[0204] In step S10, the second texture image is mapped to the second image to be pasted according to any of the above image processing methods.

[0205] In some embodiments, after mapping the first target texture image to the first image to be patched, the second texture image may be mapped to the second image to be patched according to the image processing method provided in the embodiments of the present disclosure.

[0206] In step S11, the first image to be mounted and the second image to be mounted are sequentially rendered to the target screen for display through the target rendering framework, so as to perform video texture mapping processing on the dynamic image acquired by the target device.

[0207] In some embodiments, after mapping of the first tile image or the second tile image is completed, the first tile image to be tiled may be rendered to the screen using a target rendering framework for display.

[0208] Fig.23 yes Fig. 22 Step S08 in the flowchart is shown in an exemplary embodiment. Fig.23 As shown, the above image processing method may include the following steps.

[0209] exist Fig. 22 In the method shown, due to the movement of the mobile device, the coordinate values ​​of the vertices of the target object in the image to be processed obtained by the mobile device at different times are different. Generally speaking, the target vertices can be determined from the image to be processed according to the standard graph, but this method requires re-determining the vertices of the target object in the image to be processed according to the standard graph each time. This method occupies a large amount of memory and takes a long time. Therefore, a vertex recognition method with low time complexity and low memory usage is crucial to this solution.

[0210] Since the image to be processed obtained at the current moment and the image to be processed obtained at the previous moment are essentially the same object, there is no need to redetermine the standard graph of the target object and determine the vertices of the target object based on the standard graph every time.

[0211] This embodiment provides the following method to quickly and accurately determine the vertices of the target object in the image to be processed.

[0212] In step S081, the target object position is scanned by the target device to obtain an image to be processed at the current moment.

[0213] In step S082, the image to be processed at the last moment is acquired, where the image to be processed at the last moment includes a plurality of target vertices of the target object.

[0214] In some embodiments, the target vertex information (eg, position coordinate information) in the image to be processed acquired at the last moment is generally known.

[0215] In step S083, multiple target vertices of the target object at the previous moment are processed to obtain multiple target vertices of the target object in the image to be processed at the current moment.

[0216] In some embodiments, the target vertices of the target object in the image to be processed obtained at the previous moment may be processed by an optical flow tracing method with low time complexity to determine multiple target vertices of the target object in the image to be processed at the current moment.

[0217] The optical flow tracing algorithm is an algorithm with low time complexity and has high application value for application scenarios with high real-time requirements.

[0218] The following is a brief description of the optical flow tracing algorithm.

[0219] First, based on the general assumption of optical flow tracking, that is, the pixel brightness of the feature point does not change with time, the grayscale value of the feature point does not change with time when converted to computer vision. So we get the equation:

[0220] I(x+Δx,y+Δy,t+Δt)=I(x,y,t) (7)

[0221] Where I(x, y, t) represents the grayscale value at the position (x, y) at time t, and Δx, Δy, Δt represent the offset of x, y, t.

[0222] Taylor expansion on the left, eliminating repeated factors and dividing by the time increment can obtain formulas (8) to (10) in turn:

[0223]

[0224] I is the light intensity or grayscale value at time (x, y). Vx, Vy are the moving speeds of the corresponding point (x, y) on the x-axis and y-axis on the image. Ix, Iy, It are the partial derivatives of I on each axis. After processing formula (10), we can get formula (10):

[0225] I x V x +I y V y = -I t (11)

[0226] This equation is called the optical flow equation, but due to insufficient conditions, this equation has no definite solution.

[0227] The optical flow tracking method assumes that the displacement of the image content of two adjacent frames is very small, and the displacement is roughly constant in the neighborhood of the studied point p. Therefore, it can be assumed that the optical flow equation is valid for all pixels in the window centered at point p. Apply the optical flow equation to an area near the feature point, and obtain enough equations, which can be written in the form of a matrix:

[0228]

[0229] The number of unknown variables in the equation is less than the number of equations, so the equation has a solution. The least square method (least squares method) can be used to obtain an approximate solution and obtain the values ​​of Vx and Vy, which are the two-dimensional displacement vectors of the feature points (x, y) at time t.

[0230] With an appropriate amount of two-dimensional displacement, the target vertex information of the target object at the current moment can be determined based on multiple target vertices of the image to be processed at the previous moment.

[0231] In step S084, the second image to be mounted is determined in the image to be processed at the current moment according to a plurality of target vertices of the target object at the current moment.

[0232] In some real-time processes, the second image to be mounted may be determined according to the target vertex of the target object in the image to be processed at the current moment.

[0233] Fig.24 FIG. 1 is an image processing system according to an exemplary embodiment. Fig.24As shown, after obtaining the first frame of the image to be pasted issued by the image recognition service, the system does not need to use the standard image to determine the vertices of the next image to be pasted. It only needs to use the optical flow tracking algorithm to process the vertices of the previous frame to obtain the vertex information in the current frame, and then the image segmentation processing can be performed according to the vertices of the current frame (for example, a parallelogram) to map the texture image to the image to be pasted. The technical solution provided in this embodiment uses the optical flow tracking method to track the vertices of the previous frame to determine the vertex information of the next frame. Compared with determining the vertex information of each frame according to the standard image, it can greatly improve the mapping speed and save CPU computing resources.

[0234] Fig.25 An image processing system is shown according to an exemplary embodiment. Fig.25 In this system, the target object 2501 scans the painting through the target device 2502 to obtain the image to be processed (it can be understood that the image to be processed includes not only the target object but also the background information); the target device 2502 uploads the image to be processed to the server 2503, so that the server determines the target object image from the image to be processed; the server 2503 returns the texture map and the vertex position information in the image to be pasted and the texture map to the target device 2502; the target device 2502 directly maps the texture image to the image to be pasted through the image processing method provided by the present disclosure, and renders the image to be pasted onto the screen of the target device 2502 through the target rendering framework.

[0235] Compared to Figure 4 The image processing system shown in the figure, the image processing system provided by this embodiment, does not need to use the PNP algorithm to obtain the MVP matrix between the texture image and the image to be patched, which not only simplifies the steps in the augmented reality scene, but also greatly saves CPU computing resources. This solution can transfer most of the GPU computing loss (i.e., the CPU loss caused by determining the MVP matrix) to the GPU rendering loss, so as to avoid frame drops or frame drags caused by insufficient CPU resources during the image on-screen rendering process.

[0236] Fig.26 is a block diagram of an image processing device according to an exemplary embodiment. Fig.26 The image processing device 2600 provided in the embodiment of the present disclosure may include: an image acquisition module 2601, a first segmentation module 2602, a second segmentation module 2603 and a mapping module 2603.

[0237] Among them, the image acquisition module 2601 can be configured to acquire a first image to be patched and a first texture image, wherein the first image to be patched and the first texture image are both N-gons, where N is a positive integer greater than 2; the first segmentation module 2602 can be configured to perform segmentation processing on the first image to be patched to obtain a first n-gon of the first image to be patched, where n is a positive integer greater than 2; the second segmentation module 2603 can be configured to perform segmentation processing on the first texture image to obtain a second n-gon of the first texture image, where the first n-gon and the second n-gon have a first target mapping relationship; the mapping module 2604 can be configured to map each second n-gon to the corresponding first n-gon according to the first target mapping relationship, so as to perform texture mapping processing on the first image to be patched.

[0238] In some embodiments, the first n-gon is a first triangle, the first image to be patched includes a target vertex, and the first segmentation module 2602 may include: a first dividing unit and a first connecting unit.

[0239] Among them, the first dividing unit can be configured to divide each edge of the first image to be pasted into m equal parts to obtain m equally divided points on each edge, where m is a positive integer greater than 1; the first connecting unit can be configured to connect the m equally divided points on each edge of the first image to be pasted with the target vertex to obtain the first n-gon.

[0240] In some embodiments, the first n-gon is a first triangle, the first image to be pasted is a quadrilateral, and the quadrilateral includes a target opposite side. The first segmentation unit may further include: a second segmentation unit, a second connection unit, and a diagonal connection unit.

[0241] Among them, the second segmentation unit can be configured to divide the target opposite sides of the first image to be pasted into m equal parts, and obtain m equally divided points on the target opposite sides, where m is a positive integer greater than 1; the second connection unit can be configured to connect the m equally divided points on the target opposite sides of the first image to be pasted correspondingly, so as to obtain multiple target quadrilaterals; the diagonal connection unit can be configured to connect the diagonals of the multiple target quadrilaterals, so as to obtain the first triangle.

[0242] In some embodiments, the image acquisition module 2601 may include: an image acquisition unit, a standard image determination unit, a target feature point determination unit, a first sequential connection unit, and a first to-be-mounted image determination unit.

[0243] Among them, the image acquisition unit can be configured to acquire an image to be processed including a target object; the standard image determination unit can be configured to determine a standard image matching the target object, the standard image including vertex feature points; the target feature point determination unit can be configured to determine target feature points matching the vertex feature points of the standard image in the image to be processed; the first sequential connection unit can be configured to connect the target feature points in sequence to determine the image of the target object in the image to be processed; the first image to be pasted determination unit can be configured to determine the first image to be pasted based on the image of the target object.

[0244] In some embodiments, the standard image determination unit may include: a feature extraction subunit and a feature matching subunit.

[0245] Among them, the feature extraction subunit can be configured to perform feature extraction processing on the image to be processed to obtain feature points of the image to be processed; the feature matching subunit can be configured to determine a standard image that matches the image to be processed based on the feature points of the image to be processed.

[0246] In some embodiments, the first to-be-mounted image determining unit may include: a background adding subunit.

[0247] The background adding subunit may be configured to add a background to the target object to generate the first image to be mounted, and the target object is located at a preset position of the first image to be mounted.

[0248] In some embodiments, the image acquisition module 2601 may further include: a to-be-processed image acquisition unit, an image recognition unit, and a first to-be-mounted image determination unit.

[0249] Among them, the image to be processed acquisition unit can be configured to acquire the image to be processed including the target object; the image recognition unit can be configured to determine the image of the target object in the image to be processed by image recognition technology; the first image to be pasted determination unit can be configured to determine the first image to be pasted based on the image of the target object.

[0250] In some embodiments, the image processing device 2600 may also be an image rendering device, wherein the image rendering device may be configured to render the mapped first n-gon through a target rendering framework to display it on a target screen.

[0251] In some embodiments, the image processing device 2600 may further include: a second texture image acquisition module, a texture image segmentation module and a first mapping module.

[0252] Among them, the second texture image acquisition module can be configured to acquire a second texture image, and the second texture image and the first texture image are both from a target texture video; the texture image segmentation module can be configured to segment the second texture image to obtain a third n-gon of the second texture image, and the first n-gon and the third n-gon have a second target mapping relationship; the first mapping module can be configured to map each third n-gon to the corresponding first n-gon according to the second target mapping relationship, so as to perform video texture mapping processing on the first image to be patched.

[0253] In some embodiments, the image processing device 2600 may further include: a second to-be-mounted image acquisition module, a second texture image determination module, a second mapping module, and a second rendering module.

[0254] Among them, the second image to be mounted acquisition module can be configured to acquire the second image to be mounted, the first image to be mounted and the second image to be mounted are both collected by the target device, and the second image to be mounted is the next frame image of the first image to be mounted; the second texture image determination module can be configured to acquire the second texture image, the second texture image and the first texture image are both from the target texture video, and the second texture image is the next frame image of the first texture image; the second mapping module can be configured to map the second texture image to the second image to be mounted according to the above-mentioned image processing method; the second rendering module can be configured to render the mapped first image to be mounted and the second image to be mounted in sequence to the target screen for display through the target rendering framework, so as to perform video texture mapping processing on the dynamic image acquired by the target device.

[0255] In some embodiments, the second to-be-mounted image acquisition module may include: an image acquisition unit, a previous moment target vertex acquisition unit, a current moment target vertex acquisition unit, a target object determination unit, and a second to-be-mounted image acquisition unit.

[0256] Among them, the image acquisition unit can be configured to acquire an image of the target object through the target device to obtain the image to be processed at the current moment; the target vertex acquisition unit at the previous moment can be configured to acquire the image to be processed at the previous moment acquired by the target device, and the image to be processed at the previous moment includes the vertex feature point information of the target object; the target vertex acquisition unit at the current moment can be configured to process the vertex feature point information of the target object to be processed at the previous moment, and acquire the vertex feature point information of the target object in the image to be processed at the current moment; the target object determination unit can be configured to determine the image of the target object at the current moment in the image to be processed at the current moment according to the vertex feature point information of the target object at the current moment; the second image to be pasted acquisition unit can be configured to determine the second image to be pasted according to the image of the target object at the current moment.

[0257] Since the functional modules of the image processing apparatus 2600 of the exemplary embodiment of the present disclosure correspond to the steps of the exemplary embodiment of the above-mentioned image processing method, they will not be described in detail here.

[0258] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for a computing device (which can be a personal computer, a server, a mobile terminal, or a smart device, etc.) to execute the method according to the embodiment of the present disclosure, for example Figure 5 One or more of the steps shown.

[0259] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0260] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not applied for by the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0261] It should be understood that the present disclosure is not limited to the detailed structures, drawings or implementations shown herein, but rather the present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An image processing method, characterized in that: include: Obtain a first image to be mounted and a first texture image, wherein both the first image to be mounted and the first texture image are N-gons, where N is a positive integer greater than 2; Segmenting the first image to be mounted to obtain a first n-gon of the first image to be mounted, where n is a positive integer greater than 2; Segmenting the first texture image to obtain a second n-gon of the first texture image, wherein the first n-gon and the second n-gon have a first target mapping relationship; Mapping each second n-gon to a corresponding first n-gon according to the first target mapping relationship, so as to perform texture mapping processing on the first image to be pasted; Acquire a second image to be mounted, where the first image to be mounted and the second image to be mounted are both acquired by a target device, and the second image to be mounted is a next frame image of the first image to be mounted; Acquire a second texture image, where the second texture image and the first texture image are both from a target texture video, and the second texture image is a next frame image of the first texture image; Segmenting the second image to be mounted to obtain a second n-gon of the second image to be mounted, where n is a positive integer greater than 2; Segmenting the second texture image to obtain a second n-gon of the second texture image, wherein the second n-gon and the second n-gon have a second target mapping relationship; Mapping each second n-gon to a corresponding second n-gon according to the second target mapping relationship, so as to perform texture mapping processing on the second image to be pasted; The first image to be pasted and the second image to be pasted are sequentially rendered to the target screen for display through the target rendering framework, so as to perform video texture mapping processing on the dynamic image acquired by the target device.

2. The method according to claim 1, characterized in that: The first n-gon is a first triangle, and the first image to be patched includes a target vertex; wherein the first image to be patched is segmented to obtain the first n-gon of the first image to be patched, including: Divide each side of the first image to be mounted into m equal parts to obtain m equal-divided points on each side, where m is a positive integer greater than 1; The m equally divided points on each side of the first image to be patched are connected to the target vertex to obtain the first n-gon.

3. The method according to claim 1, characterized in that: The first n-gon is a first triangle, the first image to be patched is a quadrilateral, and the quadrilateral includes a target opposite side; wherein the first image to be patched is segmented to obtain the first n-gon of the first image to be patched, including: Divide the target opposite sides of the first to-be-mounted image into m equal parts, and obtain m equal-divided points on the target opposite sides, where m is a positive integer greater than 1; Connecting m equally divided points on opposite sides of the target of the first image to be patched to obtain a plurality of target quadrilaterals; The diagonals of the plurality of target quadrilaterals are connected to obtain the first triangle.

4. The method according to claim 1, characterized in that: Obtaining the first image to be mounted, including: Acquire an image to be processed including a target object; Determine a standard graph matching the target object, wherein the standard graph includes vertex feature points; Determining target feature points in the image to be processed that match vertex feature points of the standard image; Connecting the target feature points in sequence to determine the image of the target object in the image to be processed; The first image to be mounted is determined according to the image of the target object.

5. The method according to claim 4, characterized in that: Determining a standard graph matching the target object, wherein the standard graph includes vertex feature points, including: Performing feature extraction processing on the image to be processed to obtain feature points of the image to be processed; A standard image matching the image to be processed is determined according to the feature points of the image to be processed.

6. The method according to claim 1, characterized in that: The first image to be mounted is obtained, further comprising: Acquire an image to be processed including a target object; Determine the image of the target object in the image to be processed by using image recognition technology; The first image to be mounted is determined according to the image of the target object.

7. The method according to claim 4 or 6, characterized in that: Determining the first image to be mounted according to the image of the target object includes: A background is added to the image of the target object to generate the first image to be mounted, and the target object is located at a preset position of the first image to be mounted.

8. The method according to claim 1, characterized in that: Also includes: The mapped first n-gon is rendered through a target rendering framework to be displayed on a target screen.

9. The method according to claim 1, characterized in that: Also includes: Acquire a second texture image, where the second texture image and the first texture image are both from a target texture video; Segmenting the second texture image to obtain a third n-gon of the second texture image, wherein the first n-gon and the third n-gon have a second target mapping relationship; Map each third n-gon to the corresponding first n-gon according to the second target mapping relationship, so as to perform video texture mapping processing on the first image to be pasted.

10. The method according to claim 9, characterized in that: Obtaining a second image to be mounted, including: Capturing an image of the target object through the target device to obtain an image to be processed at the current moment; Acquire the image to be processed at the last moment captured by the target device, wherein the image to be processed at the last moment includes vertex feature point information of the target object; Processing the vertex feature point information of the target object at the previous moment to obtain the vertex feature point information of the target object in the image to be processed at the current moment; Determine the image of the target object at the current moment in the image to be processed at the current moment according to the vertex feature point information of the target object at the current moment; The second image to be mounted is determined according to the image of the target object at the current moment.

11. An image processing device, characterized in that: include: An image acquisition module is configured to acquire a first image to be mounted and a first texture image, wherein both the first image to be mounted and the first texture image are N-gons, where N is a positive integer greater than 2; A first segmentation module is configured to perform segmentation processing on the first image to be mounted to obtain a first n-gon of the first image to be mounted, where n is a positive integer greater than 2; A second segmentation module is configured to perform segmentation processing on the first texture image to obtain a second n-gon of the first texture image, wherein the first n-gon and the second n-gon have a first target mapping relationship; A mapping module, configured to map each second n-gon to a corresponding first n-gon according to the first target mapping relationship, so as to perform texture mapping processing on the first image to be mounted; A second image to be mounted acquiring module, configured to acquire a second image to be mounted, wherein the first image to be mounted and the second image to be mounted are both acquired by a target device, and the second image to be mounted is a next frame image of the first image to be mounted; A second texture image determination module is configured to obtain a second texture image, wherein the second texture image and the first texture image are both from a target texture video, and the second texture image is a next frame image of the first texture image; A second mapping module is configured to segment the second image to be mounted to obtain a second n-gon of the second image to be mounted, where n is a positive integer greater than 2; Segmenting the second texture image to obtain a second n-gon of the second texture image, wherein the second n-gon and the second n-gon have a second target mapping relationship; Mapping each second n-gon to a corresponding second n-gon according to the second target mapping relationship, so as to perform texture mapping processing on the second image to be pasted; The second rendering module is configured to render the first image to be mounted and the second image to be mounted in sequence to the target screen for display through the target rendering framework, so as to perform video texture mapping processing on the dynamic image obtained by the target device.

12. The device according to claim 11, characterized in that: The first n-gon is a first triangle, and the first image to be patched includes a target vertex; wherein the first segmentation module includes: A first dividing unit is configured to divide each side of the first to-be-mounted image into m equal parts to obtain m equal division points on each side, where m is a positive integer greater than 1; The first connecting unit is configured to connect the m equally divided points on each side of the first image to be pasted with the target vertex to obtain the first n-gon.

13. The device according to claim 11, characterized in that: The first n-gon is a first triangle, the first image to be patched is a quadrilateral, and the quadrilateral includes a target opposite side; wherein the first segmentation unit further includes: A second segmentation unit is configured to divide the target opposite sides of the first to-be-mounted image into m equal parts, and obtain m equal-divided points on the target opposite sides, where m is a positive integer greater than 1; A second connection unit is configured to connect the m equally divided points on the opposite sides of the target of the first image to be pasted to obtain a plurality of target quadrilaterals; The diagonal connection unit is configured to connect the diagonals of the plurality of target quadrilaterals to obtain the first triangle.

14. The device according to claim 11, characterized in that: The image acquisition module comprises: An image acquisition unit, configured to acquire an image to be processed including a target object; A standard graph determining unit, configured to determine a standard graph matching the target object, wherein the standard graph includes vertex feature points; a target feature point determination unit, configured to determine, in the image to be processed, target feature points that match the vertex feature points of the standard image; A first sequential connection unit is configured to connect the target feature points in sequence to determine the image of the target object in the image to be processed; The first to-be-mounted image determining unit is configured to determine the first to-be-mounted image according to the image of the target object.

15. The device according to claim 14, characterized in that: The standard graph determining unit comprises: A feature extraction subunit is configured to perform feature extraction processing on the image to be processed to obtain feature points of the image to be processed; The feature matching subunit is configured to determine a standard image matching the image to be processed according to feature points of the image to be processed.

16. The device according to claim 11, characterized in that: The image acquisition module also includes: A to-be-processed image acquisition unit, configured to acquire an to-be-processed image including a target object; An image recognition unit, configured to determine the image of the target object in the image to be processed by using an image recognition technology; The first to-be-mounted image determining unit is configured to determine the first to-be-mounted image according to the image of the target object.

17. The device according to claim 14 or 16, characterized in that: Determining the first image to be mounted according to the image of the target object includes: A background is added to the image of the target object to generate the first image to be mounted, and the target object is located at a preset position of the first image to be mounted.

18. The device according to claim 11, characterized in that: The image processing device further comprises: The image rendering device is configured to render the mapped first n-gon through a target rendering framework to display it on a target screen.

19. The device according to claim 11, characterized in that: The image processing device further comprises: A second texture image acquisition module is configured to acquire a second texture image, wherein the second texture image and the first texture image are both from a target texture video; a texture image segmentation module configured to perform segmentation processing on the second texture image to obtain a third n-gon of the second texture image, wherein the first n-gon and the third n-gon have a second target mapping relationship; The first mapping module is configured to map each third n-gon to a corresponding first n-gon according to the second target mapping relationship, so as to perform video texture mapping processing on the first image to be pasted.

20. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 10.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images