Intelligent sticker generation method for vehicle and intelligent sticker system
By acquiring a vehicle space model and utilizing image capture devices and neural network technology, the problem of inaccurate positioning of in-vehicle smart stickers has been solved, achieving accurate coverage and avoiding object omissions, thus improving the user's visual experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ARCSOFT CORP LTD
- Filing Date
- 2022-04-26
- Publication Date
- 2026-06-26
AI Technical Summary
Existing in-vehicle smart sticker technology lacks a precise positioning system, resulting in inaccurate coverage areas, potential omission of object details, and impact on the user's immersive experience.
By acquiring model data of the vehicle body space, using image capturing devices to locate the interior area, and combining neural network-like technology and intelligent segmentation technology, non-overlapping intelligent sticker layers are generated to avoid covering object details.
It achieves precise positioning of smart stickers, ensuring that object details are not covered, improving the user's visual experience and providing an immersive virtual environment.
Smart Images
Figure CN117014734B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method and system for generating smart stickers for vehicles, and particularly to a method and system for generating smart stickers with real-time segmentation technology to isolate at least one object covered by the sticker. Background Technology
[0002] With the rapid advancement of technology, in-vehicle intelligence is becoming increasingly prevalent. Many application functions favored by end users can now be implemented on in-vehicle systems. For example, smart stickers are a frequently used feature on handheld mobile devices. Furthermore, many car manufacturers are considering applying smart stickers to their in-vehicle systems. Unlike handheld mobile devices, in-vehicle systems have a defined cabin space. Currently, applications with smart sticker functionality primarily utilize features adapted from handheld mobile devices, such as facial decorations, clothing decorations, and hair decorations. These applications simply overlay the smart sticker layer onto a specific area. Therefore, in the output images of traditional smart sticker applications, certain details or objects may be obscured by the smart sticker, and due to the lack of a reliable positioning system, the area covered by the smart sticker may be inaccurate.
[0003] Therefore, designing a smart sticker technology that can accurately cover the cabin space of a vehicle without missing any details and give users an immersive experience is an important issue. Summary of the Invention
[0004] This invention proposes a method for generating smart stickers for vehicles. The method includes acquiring model data of the vehicle body space, locating at least one region within the vehicle body space using at least one image capturing device, tracking at least one object within the vehicle body space to locate the image of the at least one object, and generating a layer of smart sticker based on the model data of the vehicle body space, the location data of the at least one region within the vehicle body space, and the image of the at least one object. The range of the smart sticker layer corresponds to the range of the image of the at least one object, and the smart sticker layer and the image of the at least one object do not overlap.
[0005] Another embodiment of the present invention provides a smart sticker system for vehicles. The smart sticker system for vehicles includes at least one image capturing device, a storage module, a sharing module, an output module, and a processor. At least one image capturing device is used to acquire image information streams. The storage module is used to store data. The sharing module is coupled to the storage module and is used to share data via a network. The output module is coupled to the sharing module and is used to output the processed image information stream. The processor is coupled to at least one image capturing device and the storage module and is used to process the image information stream. After acquiring model data of the vehicle body space, the processor uses at least one image capturing device to locate at least one area within the vehicle body space. The processor tracks at least one object within the vehicle body space to locate the image of at least one object. Based on the model data of the vehicle body space, the location data of at least one area within the vehicle body space, and the image of at least one object, the processor generates a layer of smart stickers and processes the image information stream accordingly. The range of the smart sticker layer corresponds to the range of the image of at least one object, and the smart sticker layer and the image of at least one object do not overlap. Attached Figure Description
[0006] Figure 1 This is an architectural diagram of an embodiment of the intelligent sticker system for vehicles according to the present invention.
[0007] Figure 2 yes Figure 1 A schematic diagram illustrating the integration of smart stickers into a static image information stream in a smart sticker system for in-vehicle use.
[0008] Figure 3 yes Figure 1 A schematic diagram illustrating the integration of smart stickers into a dynamic image information stream within a smart sticker system for in-vehicle use.
[0009] Figure 4 yes Figure 1 A flowchart illustrating the method for generating smart stickers in a vehicle-mounted smart sticker system.
[0010] The reference numerals in the attached figures are explained as follows:
[0011] 100 Smart Sticker Systems for Vehicles
[0012] 10 Image capturing device
[0013] 11 Storage Module
[0014] 12 Sharing Modules
[0015] 13 Output Module
[0016] 14 processors
[0017] 14a Image Quality Adjustment Module
[0018] 14b Distortion Correction Module
[0019] Model data of the 14c vehicle body space
[0020] 14D preprocessing module
[0021] 14e Post-processing Module
[0022] Areas C1 to C4
[0023] H1 to H4 objects
[0024] Steps S401 to S404 Detailed Implementation
[0025] Figure 1This is an architectural diagram of an embodiment of the in-vehicle smart sticker system 100 of the present invention. For simplicity of description, the in-vehicle smart sticker system 100 is referred to hereafter as the smart sticker system 100. The smart sticker system 100 includes at least one image capturing device 10, a storage module 11, a sharing module 12, an output module 13, and a processor 14. The image capturing device 10 is used to acquire image information streams. The image capturing device 10 may be a camera lens, a video recorder, or an image recorder, etc. The image information stream may include still photos or moving video data. The smart sticker system 100 does not limit the number or function of the image capturing devices 10. The storage module 11 is used to store data, such as storing images of completed stickers. The storage module 11 may be memory, a hard drive, or cloud server space, etc. The sharing module 12, coupled to the storage module 11, is used to share data via a network. For example, the sharing module 12 can share data via a vehicle-to-everything (TBox), Wi-Fi, or Bluetooth protocol. Output module 13, coupled to sharing module 12, is used to output the processed image information stream. Output module 13 can be a cloud-based mobile device terminal. Processor 14, coupled to image capturing device 10 and storage module 11, is used to process the image information stream. Processor 14 can be a vehicle infotainment system-on-chip (SoC) or any programmable processing device. In the smart sticker system 100, after processor 14 obtains model data 14c of the vehicle body space, the processor can use image capturing device 10 to locate at least one area within the vehicle body space. Processor 14 can track at least one object within the vehicle body space to locate the image of at least one object. Processor 14 can generate a layer of smart sticker based on the model data 14c of the vehicle body space, the location data of at least one area within the vehicle body space, and the image of at least one object. Furthermore, processor 14 can process the image information stream accordingly, mapping the range of the smart sticker layer to the range of the image of at least one object. The layer of the smart sticker and the image of at least one object do not overlap. In other words, the smart sticker system 100 can precisely apply smart stickers to specific areas while avoiding at least one object (such as a human figure). Therefore, the smart sticker system 100 can simulate an immersive virtual environment, enhancing the user's visual experience. The method by which the smart sticker system 100 generates smart stickers will be detailed later.
[0026] In the intelligent sticker system 100, the processor 14 can perform neural network operations, thus enabling artificial intelligence technologies such as deep learning and machine learning. The intelligent sticker system 100 can input model data 14c of the vehicle body space into the neural network to train it. After training, the neural network can be used to identify at least one region within the vehicle body space. For example, the neural network can be used to identify glass material areas such as windows, windshields, and rearview mirrors within the vehicle body space. The intelligent sticker system 100 can also allow the user to specify at least one region. For example, the user can specify glass material areas such as windows, windshields, and rearview mirrors within the vehicle body space. Similarly, after training, the neural network can be used to identify at least one region within the vehicle body space. The neural network can also utilize intelligent segmentation technology to determine at least one region within the vehicle body space. For example, the neural network can use intelligent segmentation technology to segment glass material areas such as windows, windshields, and rearview mirrors within the vehicle body space. Any reasonable technical modifications fall within the scope of this invention. The aforementioned use of neural networks to identify at least one region within the vehicle body space constitutes a coarse localization of at least one region. In the smart sticker system 100, the image capturing device 10 can detect multiple vertices of at least one region. For example, the image capturing device 10 can detect multiple vertices of a polygonal vehicle window. The processor 14 can locate at least one region based on the multiple vertices of the at least one region. For example, the processor 14 can detect the three-dimensional coordinates of multiple vertices within the vehicle body space. Therefore, using information from the image capturing device 10 to locate at least one region constitutes a precise localization of at least one region.
[0027] Please refer to Figure 1. After the image capturing device 10 acquires the image information stream, it can transmit the image information stream to the processor 14 via a communication protocol or transmission channel. The processor 14 includes an image quality adjustment module 14a, a distortion correction module 14b, vehicle space model data 14c, a pre-processing module 14d, and a post-processing module 14e. After receiving the image information stream, the processor 14 can use the image quality adjustment module 14a to adjust the image's brightness, white balance, saturation, contrast, noise, etc. Then, the image with image quality adjustment is transmitted to the distortion correction module 14b. The distortion correction module 14b can adjust the image distortion, such as wide-angle distortion or tubular distortion, etc. The vehicle space model data 14c may include three-dimensional vehicle space model data, which can be pre-stored in the memory of the processor 14. The pre-processing module 14d can perform image processing on the image with image quality adjustment and distortion correction based on the vehicle space model data 14c, as explained below. For simplicity, at least one object within the vehicle space is depicted as a human figure. The pre-processing module 14d performs object segmentation and human figure segmentation. As mentioned earlier, object segmentation identifies regions within the vehicle space, separating glass-covered areas from other areas. The underlying object segmentation technology is based on deep learning, optimized for deployment scenarios and platform characteristics, particularly in network structure, training data, and learning strategies, achieving accurate and stable segmentation. Furthermore, object segmentation can be combined with network storage (NAS), knowledge distillation techniques, and in-depth optimization of the underlying code, enabling efficient algorithm operation. In human figure segmentation, the pre-processing module 14d detects the outline of at least one human figure within the vehicle space. Then, based on the outline of the at least one human figure, the pre-processing module 14d separates the image of the at least one human figure from the background. The image of the at least one human figure is also marked by the processor 14, ensuring that the smart sticker layer is mapped to at least one region during subsequent processing, preventing the smart sticker layer from overlapping with the image of the at least one human figure.
[0028] After the preprocessing module 14d within the processor 14 performs human and object segmentation techniques, the image data is transmitted to the postprocessing module 14e. The postprocessing module 14e performs perspective distortion fitting on the smart sticker image according to the identified glass area, as detailed below. First, the postprocessing module 14e detects multiple vertices of at least one segmented area. Next, the postprocessing module 14e rotates the smart sticker layer based on the position and angle of at least one image capturing device to map the smart sticker layer onto at least one area. As mentioned earlier, at least one area can be a glass area. Furthermore, at least one area can be generated using glass segmentation technology. In the smart sticker system 100, the underlying glass segmentation technology can be based on deep learning technology using artificial intelligence, utilizing a fixed in-vehicle camera to collect a large number of different scene samples, and labeling the ground truth of the car window glass area according to custom rules. Glass segmentation technology can also employ the U-Net deep learning model, utilizing the normalized exponential function (Softmax) cross-entropy to calculate the loss optimization network with a ground truth mask, combined with post-processing optimization to obtain the final masking result. The post-processing module 14e can also analyze the image information stream to map the smart sticker layer to at least one region, ensuring that the smart sticker layer does not overlap with at least one human figure image. In other words, the post-processing module 14e can utilize human face tracking technology to dynamically segment moving human figures in real time and protect them. Glass-covered areas or other vehicle body areas obscured by the human figure will not be covered with stickers. Furthermore, the smart sticker layer is applied to a two-dimensional planar pattern as a single-frame static image or as a sequence of dynamic frames in a two-dimensional planar pattern. Next, the processed image information stream can be transmitted to the storage module 11. The processed image information stream includes image data after sticker synthesis processing. The storage module 11 can store the image data after sticker synthesis processing. The processed image stream can be further transmitted to the sharing module 12. Therefore, the sharing module 12 can share the image data after sticker compositing to the network. Finally, the processed image stream can be transmitted to the output module 13. Users can view the image data after sticker compositing through the output module 13.
[0029] For ease of understanding, the following embodiment describes the actual implementation method of generating smart stickers with human figures and multiple glass areas within the vehicle space. First, the position of the image capturing device 10 inside the vehicle is determined, and the position of the image capturing device 10 and the imaging position of the glass areas within the image capturing device 10 are calibrated. Next, based on the calibration parameters, the positions of all glass areas, including the windshield, rear windshield, sunroof, and windows, are determined. Next, using the aforementioned glass segmentation technology, a layer mask is obtained. Next, the processor 14 can detect faces. If a face is detected, portrait segmentation is performed, and the portrait layer is protected to prevent the smart sticker from covering it. If no face is detected, the processor 14 can detect a human body. If a human body is detected, portrait segmentation is performed, and the portrait layer is protected to prevent the smart sticker from covering it. In other words, if either a face or a human body is detected by the processor 14, the portrait segmentation procedure is executed. If neither a face nor a human body is detected, it indicates that no one is inside the vehicle, and the processor 14 can execute an object detection procedure. If a specific object is detected, the processor 14 can execute the aforementioned object segmentation program, using deep learning technology to segment objects within the vehicle body. For example, it can identify foreground objects on the rear windshield to avoid covering them with the smart sticker. If no specific object is detected, it means there is no one inside the vehicle and no object image to process; therefore, the processor 14 can directly apply the smart sticker to the glass area. The above embodiments are merely one way in which the smart sticker system 100 of the present invention generates smart stickers, and the operation of the smart sticker system 100 is not limited to the above embodiments.
[0030] Figure 2 This is a schematic diagram illustrating the integration of smart stickers into a static image information stream in a smart sticker system 100 for use in vehicles. Figure 3 This is a schematic diagram illustrating the integration of smart stickers into a dynamic image information stream within a smart sticker system 100 for in-vehicle use. As mentioned earlier, the smart sticker layer can be applied to a two-dimensional planar graphic in the form of a single frame static image, or it can be applied to a two-dimensional planar graphic in the form of a sequence of dynamic frames. Figure 2In the image, objects H1 and H2 are two static human figures. Regions C1, C2, C3, and C4 are glass areas. For example, region C1 could be a sunroof, region C2 could be the right window, region C3 could be the left window, and region C4 could be the rear windshield. For the static image information stream, the smart sticker is in the form of a single-frame static image. Processor 14 can detect the two static human figures (objects H1 and H2). Processor 14 can also use object detection technology to detect all foreground objects in regions C1, C2, C3, and C4, such as seats, rearview mirrors, steering wheels, etc. Finally, processor 14 can map the smart sticker to the glass areas of regions C1, C2, C3, and C4. Furthermore, the coverage area of the smart sticker does not overlap with the two static human figures (objects H1 and H2) and all foreground objects. Similarly, in Figure 2 In this system, objects H1 and H2 are two dynamic human figures. For the dynamic image information stream, the smart sticker takes the form of a sequence of dynamic frames. Processor 14 can detect the two dynamic human figures (objects H1 and H2). Processor 14 can also use object detection technology to detect all foreground objects in regions C1, C2, C3, and C4, such as seats, rearview mirrors, steering wheels, etc. Finally, processor 14 can map the smart sticker to the glass areas of regions C1, C2, C3, and C4. Furthermore, the coverage area of the smart sticker does not overlap with the two dynamic human figures (objects H1 and H2) and all foreground objects. In other words, regardless of whether the image information stream is in a static or dynamic data format, the smart sticker system 100 can output images or videos that provide an immersive experience for the user.
[0031] Figure 4 This is a flowchart illustrating the smart sticker generation method of a vehicle-mounted smart sticker system 100. The process of generating the smart sticker includes steps S401 to S404. Any reasonable modifications to the steps fall within the scope of this invention. Steps S401 to S404 are as follows:
[0032] Step S401: Obtain model data 14c of the vehicle body space;
[0033] Step S402: Use at least one image capturing device 10 to locate at least one area C1 to C4 within the vehicle body space;
[0034] Step S403: Track at least one object H1 to H4 within the vehicle body space to locate the image of at least one object;
[0035] Step S404: Generate a layer of smart sticker based on the model data 14c of the vehicle body space, the positioning data of at least one area C1 to C4 within the vehicle body space, and the image of at least one object H1 to H4.
[0036] The details of steps S401 to S404 have been described in detail above, and therefore will not be repeated here. The smart stickers generated by the smart sticker system 100 based on steps S401 to S404 can not only be applied to designated areas, but also protect special objects (such as portraits or foreground objects) from being covered by the smart stickers. Therefore, the smart sticker system 100 can provide highly realistic composite images or videos, thus enhancing the user's visual experience.
[0037] In summary, this invention describes a smart sticker system and a method for generating smart stickers for in-vehicle use. The smart sticker system utilizes object segmentation technology, human image tracking technology, and human image segmentation technology to separate the glass area inside the vehicle from the human image area. Furthermore, the smart sticker system can detect and separate foreground objects within the glass area. Because the smart sticker system can identify and separate objects inside the vehicle, when a smart sticker corresponds to a specific area, it can avoid at least one object (e.g., a human figure or a foreground object), ensuring that the smart sticker layer does not overlap with the image of at least one object. Therefore, the smart sticker system of this invention can simulate an immersive virtual environment, enhancing the user's visual experience.
[0038] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for generating smart stickers for vehicles, characterized in that, include: Obtain model data of the vehicle body space; At least one area within the vehicle body space is located using at least one image capturing device; Tracking at least one object within the vehicle body space to locate an image of the at least one object; and Based on the model data of the vehicle body space, the positioning data of the at least one area within the vehicle body space, and the image of the at least one object, a layer of smart sticker is generated. The range of the layer of the smart sticker corresponds to the range of the image of the at least one object, and the layer of the smart sticker does not overlap with the image of the at least one object.
2. The method as described in claim 1, characterized in that, Also includes: Specify at least one region; and Based on the model data of the vehicle body space, recognition technology is used to identify at least one area within the vehicle body space; The at least one of the regions includes a glass material region.
3. The method as described in claim 1, characterized in that, Also includes: The model data of the vehicle body space is input into a neural network to train the neural network. and Using a trained neural network, at least one region is identified within the vehicle body space; The at least one of the regions includes a glass material region.
4. The method as described in claim 1, characterized in that, Locating at least one region within the vehicle body space using the at least one image capturing device includes: The at least one image capturing device detects multiple vertices of the at least one region; and The at least one region is located based on multiple vertices of the at least one region.
5. The method as described in claim 4, characterized in that, Also includes: The layer of the smart sticker is rotated according to the position and angle of the at least one image capturing device, so as to map the layer of the smart sticker onto the at least one area.
6. The method as described in claim 1, characterized in that, At least one object within the vehicle body space includes at least one human figure within the vehicle body space, the model data of the vehicle body space includes three-dimensional vehicle body space model data, and the layer of the smart sticker is a two-dimensional planar graphic.
7. The method as described in claim 6, characterized in that, Also includes: Detect the outline boundary of at least one human figure within the vehicle body space; Based on the outline boundary of the at least one human figure, the image of the at least one human figure is separated from the background; and When the image of the at least one human figure is marked so that the layer of the smart sticker is aligned with the at least one area, the layer of the smart sticker does not overlap with the image of the at least one human figure.
8. The method as described in claim 1, characterized in that, Also includes: Obtain the image information stream within the vehicle body space; Analyze the image information stream to map the layer of the smart sticker onto the at least one region. Furthermore, the layer of the smart sticker does not overlap with the image of the at least one human figure; and Output the processed image information stream; The image information stream includes static photos or dynamic video data, and the layer of the smart sticker is in the form of a single frame static image or a sequence of frames dynamic images.
9. The method as described in claim 1, characterized in that, Also includes: Based on the model data of the vehicle body space, intelligent segmentation technology is used to determine at least one region within the vehicle body space.
10. A smart sticker system for vehicles, characterized in that, include: At least one image capturing device for acquiring an image information stream; Storage module, used to store data; A sharing module, coupled to the storage module, is used to share data over a network; An output module, coupled to the sharing module, is used to output the processed image information stream; and A processor, coupled to the at least one image capturing device and the storage module, is used to process the image information stream; After acquiring the model data of the vehicle body space, the processor uses the at least one image capturing device to locate at least one area within the vehicle body space. The processor tracks at least one object within the vehicle body space to locate the image of the at least one object. Based on the model data of the vehicle body space, the location data of the at least one area within the vehicle body space, and the image of the at least one object, the processor generates a layer of smart stickers and processes the image information stream accordingly. The range of the smart sticker layer corresponds to the range of the image of the at least one object, and the smart sticker layer does not overlap with the image of the at least one object.