Object image rendering method and system

Abstract

The invention discloses an object image rendering method and a system thereof, which are used for solving the problem that a known image processing technology cannot present a 3D image of the complete appearance of a reproduced object. The method specifically comprises the following steps: identifying region position information, region contour and region appearance of each part region of an object of interest from an input image; establishing a contour mapping relation between the regional position information and the corresponding regional contour; establishing an appearance mapping relationship between the regional position information and the corresponding regional appearance; according to the contour mapping relation, a reproduced object contour is formed, and all areas, corresponding to all the area position information, of the reproduced object contour have corresponding area contours; and according to the appearance mapping relation, enabling the contour of the reproduced object to correspond to each region of the region position information, and rendering the corresponding region appearance. According to the invention, the effect of providing real and vivid visual experience for a driver can be achieved.

Description

Technical Field

[0001] The invention relates to a vehicle image processing technology, in particular to an object image rendering method and system thereof. Background Art

[0002] With the development of image processing technology, the application of vehicle safe driving is becoming more and more mature, especially the technology that can identify objects of interest (especially other vehicles) other than the driving vehicle from the 2D image captured by the image capture unit as the input image, and convert the 2D image into 3D position information. In addition to enabling the background computing module to more accurately determine the distance between each object and the driving vehicle to achieve safe driving warning or automatic driving technology; at the same time, the 3D position information can also be reproduced / converted into a 3D image and presented in the auxiliary driving display. In particular, since the reproduced 3D image can only present objects of interest and other important objects, and can filter out unimportant noise in the actual environment scene, it is helpful for the driver to observe the picture in the auxiliary driving display and judge the driving environment more accurately to improve driving safety.

[0003] Please refer to Figure 1-3 As shown, the first figure shows that an object of interest O' in the input image (2D image) is an image of a vehicle C'; the second figure shows information of multiple part areas PA' of the vehicle identified in the prior art, and only has the area outline AO' of the multiple part areas PA'; the third figure shows that each of the part areas PA' is reassembled into a 3D reconstructed object RO' according to their individual positional relationships. Therefore, by Figure 1-3 It can be seen that the recreated object RO' converted by the known technology only has the regional outline AO' and presents the appearance of a single color block, so it lacks the real details of the corresponding vehicle C' in appearance. As a result, when the driver watches the recreated object RO' as a 3D image, it cannot provide the driver with a more realistic and vivid visual experience, and creates a sense of disconnection with the appearance of other vehicles in the actual environment.

[0004] In view of this, the conventional image processing technology still needs to be improved. Summary of the invention

[0005] To solve the above problems, an object of the present invention is to provide an object image rendering method and system thereof, which can improve the authenticity of the appearance of a recreated 3D model of a vehicle, thereby improving the driving experience.

[0006] The directions or their approximate terms described throughout the present invention, such as "front", "rear", "left", "right", "up (top)", "down (bottom)", "inside", "outside", "side", etc., are mainly with reference to the directions of the attached drawings. Each direction or its approximate terms are only used to assist in the explanation and understanding of the various embodiments of the present invention, and are not used to limit the present invention.

[0007] The quantifiers “a” or “an” used in the elements and components described throughout the present invention are only for convenience of use and to provide a general meaning of the scope of the present invention; in the present invention, they should be interpreted as including one or at least one, and a single concept also includes plural cases, unless it is obvious that it means otherwise.

[0008] The term "coupling" as used throughout the present invention includes direct or indirect electrical and / or signal connections, which can be selected by those skilled in the art according to usage requirements.

[0009] The similar terms such as "combination", "assembly", "assembly" and "setting" mentioned throughout the present invention mainly include the connection that can be separated without destroying the components, or the connection that makes the components inseparable, and can be selected by those with ordinary knowledge in the field according to the materials of the components to be connected or the assembly requirements.

[0010] The object image rendering method of the present invention is executed through a processor, and includes: obtaining an input image, and the input image has an identifiable object of interest; identifying a region position information, a region outline and a region appearance corresponding to each of at least one region of the object of interest from the input image; in a case where the at least one region is multiple, each region is different; the region outline and the region appearance of the same region have the same region position information; establishing a contour mapping relationship between each region position information and the region outline of a corresponding one of the at least one region; establishing an appearance mapping relationship between each region position information and the region appearance of a corresponding one of the at least one region; according to the contour mapping relationship, forming a reconstructed object contour in an output image, so that the reconstructed object contour corresponds to a region of each region position information and has the contour of a corresponding one of the region outlines; and according to the appearance mapping relationship, rendering the appearance of a corresponding one of the region appearances by making the reconstructed object contour correspond to each region of each region position information.

[0011] The object image rendering system of the present invention comprises: an image capture unit for capturing an input image, wherein the input image has an identifiable object of interest; a processing unit coupled to the image capture unit to receive the input image; the processing unit having: a trained image recognition model for identifying, from the input image, a region position information, a region outline and a region appearance corresponding to each of at least one region of an object of interest; the region outline and the region appearance of the same region have the same region position information; and a trained image generation model for establishing a contour by combining each of the region position information with the corresponding one of the region areas. mapping relationship; and the image generation model establishes an appearance mapping relationship between each of the regional position information and a corresponding one of the equal-region appearances; the image generation model forms a recreated object contour in an output image according to the contour mapping relationship, so that the recreated object contour corresponds to an area of ​​each of the regional position information and has the contour of a corresponding one of the equal-region contours; the image generation model makes the recreated object contour correspond to an area of ​​each of the regional position information according to the appearance mapping relationship, renders the appearance of a corresponding one of the equal-region appearances, so as to generate an output image; and a display unit is coupled to the processing unit to receive and display the output image.

[0012] Accordingly, the object image rendering method and system of the present invention, by identifying the area position information, area contour and area appearance, and by establishing the contour mapping relationship and the appearance mapping relationship, after forming the recreated object contour, can render the corresponding appearance at the corresponding position in the recreated object contour, so that the recreated object image has an appearance close to the real thing, and has the effect of providing the driver with a real and vivid visual experience.

[0013] Wherein, the object of interest is a vehicle, and the part region includes at least one of a door, a window, a windshield, a tire, a roof, a front, a rear, a front body, and a rear body. Thus, through each identifiable part region of the object of interest, the region outline and the region appearance can be identified, so that a corresponding 3D object is established through the region outline, and then the corresponding color is rendered through the region appearance, so that the reproduced object image has a close-to-real appearance, and has the effect of providing the driver with a real and vivid visual experience.

[0014] Each of the regional appearances includes at least one of color information, texture information, and pattern information of a corresponding one of the at least one region. Thus, by rendering the corresponding color of the regional appearance, the reproduced object image can have a close-to-real appearance, providing the driver with a real and vivid visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of input image showing known image processing techniques.

[0016] Figure 2 Show Figure 1 Schematic diagram of the regional contours of various parts of the area identified by known image processing techniques.

[0017] Figure 3 Show Figure 2 Schematic diagram of a recreated object generated by a known image processing technique.

[0018] Figure 4 A system block diagram of a preferred embodiment of the object image rendering system of the present invention is shown.

[0019] Figure 5 A schematic diagram showing the object image rendering method and system of the present invention for identifying various regions of interest in an input image.

[0020] Figure 6 The present invention shows Figure 5 Schematic diagram of identifying / extracting the regional contours of various parts.

[0021] Figure 7 The present invention shows Figure 5 Schematic diagram of the regional appearance of each part area identified / extracted in .

[0022] Figure 8 The present invention is shown Figure 6 and 7 A schematic diagram of a processing flow of generating a remade object outline and then rendering the appearance of a corresponding area to generate the remade object.

[0023] Fig. 9 A flow chart showing the steps of a preferred embodiment of the object image rendering method of the present invention.

[0024] Wherein: 1: image capture unit, 2: processing unit, 3: display unit, AA: area appearance, AO: area outline, C: vehicle, D: door, FB: front body, FC: front hood, O: object of interest, PA: part area, R: roof, RB: rear body, RO: reconstructed object, ROO: reconstructed object outline, S1: image input step, S2: feature recognition and extraction step, S3: feature and position association step, S4: image reconstruction step, T: tire, W: window, WS: windshield; In the prior art: AO': area outline, C': vehicle, O': object of interest, PA': part area, RO': reproduced object. DETAILED DESCRIPTION

[0025] In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the preferred embodiments of the present invention are specifically cited below and described in detail with reference to the accompanying drawings; in addition, those marked with the same symbols in different drawings are regarded as the same and their descriptions will be omitted.

[0026] like Figure 4 As shown, a preferred embodiment of the object image rendering system of the present invention comprises an image capturing unit 1, a processing unit 2 and a display unit 3, wherein the processing unit 2 is coupled to the image capturing unit 1 and the display unit 3 respectively.

[0027] The image capture unit 1 is disposed on a driving vehicle to capture an environment image around the driving vehicle as an input image. Preferably, the input image includes an object of interest O that can be identified by an image recognition model.

[0028] The processing unit 2 is coupled to the image capture unit 1 to receive the input image. Optionally, the processing unit 2 can be set in the driving vehicle, or in a remote state (such as a cloud system). The processing unit 2 includes at least one processor, and has a trained image recognition model and a trained image generation model. In particular, the processing unit 2 is coupled to the display unit 3, and the input image is processed by the processing unit 2 to generate an output image, and the display unit 3 receives and displays the output image.

[0029] As shown in FIGS. 5 to 7, the image recognition model is used to identify a region position information, a region outline AO ​​and a region appearance AA corresponding to each of at least one region PA of an object of interest O from the input image; the region outline AO ​​and the region appearance AA of the same region PA have the same region position information. In detail, the region position information can be defined by coordinates in a coordinate system; each region appearance AA includes at least one of a color information, a texture information and a pattern information of a corresponding one of the at least one region PA.

[0030] It should be noted that the image recognition model can be implemented through technologies such as object detection and / or semantic segmentation, especially by using a multi-learning network architecture with corresponding models and algorithms, and optionally using images of labeled objects or classified features as training data for corresponding training and verification. In addition, the "recognition" of features such as the object of interest O, each part area PA, each area position information, each area outline AO ​​and each area appearance AA includes specific procedures such as extraction, storage and transmission of corresponding information.

[0031] In particular, as shown in FIG. 5 , the object of interest O may be a vehicle C, and the part area PA may include but is not limited to at least one of a door D, a window W, a windshield WS, a tire T, a roof R, a front hood FC, a front body FB, and a rear body RB. In addition, in other perspectives (not shown), the part area PA may be the front or rear of the vehicle. The door D may be a front door or a rear door; the window W may be any window corresponding to the door D; the windshield WS may be a front windshield or a rear windshield; the front body FB includes a portion from a front door D to the front of the vehicle C; and the rear body RB includes a portion from a rear door D to the rear of the vehicle C.

[0032] It should be noted that the corresponding image recognition model can be trained and established according to the actual recognition needs, so that through the impact recognition model, multiple of the multiple part areas PA can be integrated into one identifiable area, or multiple identifiable areas can be further divided from each of the multiple part areas PA; for example, in the examples shown in Figures 5 to 7 of the present invention, the front door D, the corresponding window W and the front body FB can be integrated into a identifiable part area PA; the rear door D and the corresponding window W can be integrated into another current identifiable part area PA.

[0033] As shown in FIG. 8 , the image generation model corresponds each of the regional position information to the part area PA, and establishes a contour mapping relationship; and the image generation model corresponds each of the regional position information to the regional appearance AA, and establishes an appearance mapping relationship. In the first state of FIG. 8 , the image generation model forms a reconstructed object contour ROO in an output image according to the contour mapping relationship, so that the reconstructed object contour ROO corresponds to an area of ​​each of the regional position information, and has a corresponding contour in the equal regional contour AO. In the second state of FIG. 8 , the image generation model makes the reconstructed object contour ROO correspond to an area of ​​each of the regional position information according to the appearance mapping relationship, and renders the corresponding appearance in the regional appearance AA to generate a reconstructed object RO in the output image; in the output image, the 3D image of the reconstructed object RO has the appearance of a 2D image of the object of interest O in the input image.

[0034] It should be noted that in the process of converting the captured 2D input image into the 3D output image, in addition to presenting the reproduced object RO corresponding to the object of interest O in the output image, important information during driving can also be presented, and unnecessary information can be excluded, so that the output image can present important information more concisely. For example, the input image in FIG. 5 has objects such as road markings, dividing islands, vehicles and road trees, while the output image in FIG. 8 excludes road trees.

[0035] According to the aforementioned system, as shown in FIG. 9 , the present invention can implement an object image rendering method, particularly executed by the processor in the processing unit 2, comprising an image input step S1, a feature recognition and extraction step S2, a feature and position association step S3 and an image reconstruction step S4. The above steps are described as follows: In FIG. 9 , in the image input step S1 , an input image (as shown in FIG. 5 ) is acquired, and the input image has a recognizable object of interest O.

[0036] In Figure 9, in the feature recognition and extraction step S2, a region position information, a region outline AO ​​and a region appearance AA corresponding to each of at least one part area PA of the object of interest O are identified from the input image (as shown in Figures 6 and 7); in a case where at least one of the part areas PA is multiple, each of the part areas PA is different; the region outline AO ​​and the region appearance AA of the same part area PA have the same region position information.

[0037] In the feature and position association step S3 of Figure 9, a contour mapping relationship is established between each of the regional position information and the corresponding regional contour AO in at least one part area PA; and an appearance mapping relationship is established between each of the regional position information and the corresponding regional appearance AA in at least one part area PA.

[0038] In the image reconstruction step S4 of FIG. 9, a reconstructed object outline ROO is formed in an output image according to the outline mapping relationship (as shown in the first state of FIG. 8), so that the reconstructed object outline ROO corresponds to an area of ​​each of the area position information and has a corresponding outline in each of the area outlines AO. Then, according to the appearance mapping relationship, the reconstructed object outline ROO corresponds to each of the areas of each of the area position information, and the corresponding appearance in each of the area appearances AA is rendered to generate a reconstructed object RO (as shown in the second state of FIG. 8).

[0039] In summary, the object image rendering method and system of the present invention, by identifying the area position information, area contour and area appearance, and by establishing the contour mapping relationship and the appearance mapping relationship, after forming the reconstructed object contour, can render the corresponding appearance at the corresponding position in the reconstructed object contour, so that the reconstructed object image has a close-to-real appearance, providing the driver with a real and vivid visual experience.

[0040] Although the present invention has been disclosed using the above preferred embodiments, they are not intended to limit the present invention. Any person skilled in the art may make various changes and modifications to the above embodiments without departing from the spirit and scope of the present invention. These changes and modifications are still within the technical scope protected by the present invention. Therefore, the protection scope of the present invention shall include all changes within the meaning and equivalent scope of the attached patent application.

Claims

1. An object image rendering method, executed by a processor, characterized in that: Include: Acquire an input image, wherein the input image has a recognizable object of interest; Identify from the input image a region position information, a region outline and a region appearance corresponding to each of at least a part region of the object of interest; In the case where at least one of the part regions is multiple, each of the part regions is different; the area contour and area appearance of the same part region have the same area position information; Establishing a contour mapping relationship between each of the regional position information and the corresponding regional contour in at least one of the part regions; establishing an appearance mapping relationship between each of the regional position information and the corresponding regional appearance in at least one of the part regions; According to the contour mapping relationship, a reconstructed object contour is formed in an output image, so that the reconstructed object contour corresponds to the area of ​​each of the area position information and has a corresponding contour in each of the area contours; as well as According to the appearance mapping relationship, the outline of the reproduced object is made to correspond to each of the regions of each of the region position information, and the corresponding appearance in each of the region appearances is rendered.

2. The object image rendering method according to claim 1, characterized in that: The object of interest is a vehicle, and the part area includes at least one of a door, a window, a windshield, a tire, a roof, a front, a rear, a front body and a rear body.

3. The object image rendering method according to claim 1 or 2, characterized in that: Each of the regional appearances includes at least one of color information, texture information, and pattern information corresponding to at least one of the part regions.

4. An object image rendering system, using an object impact rendering method as claimed in any one of claims 1 to 3, characterized in that: Include: An image capture unit, used to capture an input image, wherein the input image has an identifiable object of interest; A processing unit is coupled to the image capture unit to receive the input image; the processing unit has: A trained image recognition model is used to identify, from the input image, a region position information, a region outline and a region appearance corresponding to each of at least one region of an object of interest; the region outline and region appearance of the same region have the same region position information; as well as A trained image generation model places each of the regional position information and the low-frequency band into the part region to correspond to each other, and establishes a contour mapping relationship; and the image generation model corresponds each of the regional position information and the regional appearance to correspond to each other, and establishes an appearance mapping relationship; the image generation model forms a reconstructed object contour in an output image according to the contour mapping relationship, so that the reconstructed object contour corresponds to an area of ​​each of the regional position information and has a corresponding contour in the equal regional contour; the image generation model makes the reconstructed object contour correspond to an area of ​​each of the regional position information according to the appearance mapping relationship, and renders the corresponding appearance in the regional appearance to generate an output image; and A display unit is coupled to the processing unit to receive and display the output image.

5. The object image rendering system according to claim 4, characterized in that: The object of interest is a vehicle, and the part area includes at least one of a door, a window, a windshield, a tire, a roof, a front, a rear, a front body and a rear body.

6. An object image rendering system as claimed in claim 4 or 5, characterized in that: Each of the region appearances includes at least one of color information, texture information, and pattern information corresponding to at least one of the region parts.

Images