A control unit for fine-tuning an intelligent / baselined model and a method thereof
By using an expert model to generate multiple titles in the control unit and fine-tuning the intelligent model in conjunction with a large language model, the limitations of general-purpose computer vision systems are overcome, enabling efficient image search and filtering in specific fields.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2025-11-27
- Publication Date
- 2026-05-29
AI Technical Summary
Existing computer vision systems are limited by fixed, predetermined object category supervision, which restricts their versatility and usability. They require a large amount of labeled data to specify visual concepts. Learning directly from the raw text of images is an improved approach.
The control unit receives data from multiple sources, uses an expert model to identify objects and generate multiple titles, combines a large language model fusion unit and a text encoder, uses contrastive training techniques to fine-tune the intelligent model, and generates a parallel corpus to adapt to a specific domain.
It provides an accurate image search and filtering solution on unseen datasets, applicable to complex scenarios such as autonomous driving, industrial robots, and monitoring systems.
Smart Images
Figure CN122116024A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a control unit and method for fine-tuning an intelligent / basic model. More specifically, the intelligent / basic model is a visual basic model. Background Technology
[0002] Existing computer vision systems are trained to predict a fixed set of predetermined object categories. This limited form of supervision restricts their generality and usability because additional labeled data is required to specify any other visual concepts. Learning directly from raw text about images is a promising alternative that leverages a wider range of supervision sources.
[0003] Patent application US2022383048 discloses a system and method for cross-linguistic, cross-modal training for multimodal retrieval (aa). This application discloses that current pre-trained visual-language models for English cross-modal retrieval tasks rely on the availability of numerous labeled image-caption datasets for pre-training to obtain English text. The embodiments described herein discuss learning cross-linguistic, cross-modal representations for matching images and their associated captions in multiple languages. The embodiments seamlessly combine cross-linguistic and cross-modal pre-training objectives within a unified framework to learn images and text in a joint embedding space from available English image-caption data, monolingual corpora, and parallel corpora. The embodiments are demonstrated to achieve state-of-the-art performance on retrieval tasks on multimodal, multilingual image-caption datasets. Attached Figure Description
[0004] Figure 1 The illustration shows a control unit for fine-tuning an intelligent / basic model according to an embodiment of the present invention; and Figure 2 The illustration shows a method for fine-tuning an intelligent / basic model using a control unit according to the present invention. Detailed Implementation
[0005] Figure 1The illustration depicts a control unit for fine-tuning an intelligent / base model according to an embodiment of the present invention. The control unit 10 receives data from multiple sources 12 for training and developing an intelligent model 20, and then at least one expert model 15 identifies and labels each object present in each data point 11 of the received data. The control unit 10 then derives metadata about each of the at least one expert model 15 and generates multiple titles using a large language model fusion processor 18. The control unit 10 sends the generated multiple titles associated with each data point 11 to a text encoder 16, which maps at least one generated title to a corresponding data point 11. The control unit 10 then uses a contrastive training technique to fine-tune the intelligent model 20 after mapping the data points 11 to the corresponding at least one generated title.
[0006] Furthermore, the arrangement of the components of the control unit and its operation are explained in detail. Control unit 10 is an operational device for processing and storing information received from at least one source. Control unit 10 is selected from the group of control units including microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, one or more microchips or integrated circuits interconnected using a motherboard, hard-wired logic, software stored in a memory device and executed by a microprocessor, firmware, application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs), and / or any component that operates on signals based on operating instructions. It should be noted that the control unit can be any other type, but is not limited to the devices described above known to those skilled in the art.
[0007] The control unit 10 is connected via the cloud or any other communication means known in the art. The control unit 10 includes multiple hardware units, such as a large language model fusion unit 18, expert models 15 each designed to perform corresponding functions, a memory (not shown) for storing received data, encoders (14 and 16) for mapping functions, and an intelligent model 20 with a neural network. These units are interconnected in some way such that the intelligent model 20 present in the control unit 10 is either trained and developed from scratch or fine-tuned. According to one embodiment of the invention, the intelligent model 20 is a visual foundation model. Many foundation models 20 are commercially available; one such foundation model is the CLIP model, which is used for complex image search.
[0008] These base models 20 are trained using a large amount of data received from multiple sources 12. The received data is not domain-specific, and these base models 20 cannot operate efficiently when deployed in domain-specific operations. To overcome this, the base models 20 are trained using a text corpus related to the images present in the data. By leveraging this fine-tuning of the base / intelligent models 20, complex image searches can be easily achieved. It should be noted that this invention is not merely about the computer itself, but rather an invention of means + function, wherein the control unit 10 is provided with certain inputs to obtain a desired output, which involves the hardware units described above.
[0009] According to one embodiment of the invention, the data is a set of images of different scenes involving multiple objects. Each image includes multiple objects captured at a given point in time. The objects in this document are related to things and people captured in that particular scene. Each data point 11 of the data represents an image from the set of received images. The control unit 10 includes multiple expert models 15. The control unit 10 may contain from a minimum of one expert model 15 to "n" expert models 15. Each expert model 15 performs a specific function for providing a label for each object present in each data point / image 11.
[0010] Labels are provided as tokens for each object present in each data point / image 11. It should be noted that, at any point in this specification, data point 11 and image 11 should be considered the same. Expert models 15 are selected from a set of expert models including zero-shot labeling expert model 15(a), semantic segmentation expert model 15(c), and automatic labeling expert model 15(b). Each expert model 15 is associated with the attributes of objects in each data point 11; one such expert model 11 (automatic labeling expert model 15(b)) is used to identify all road signs, and another expert model 15(a) is associated with the entire scene and is used to identify the weather in each data point 11.
[0011] The control unit 10 generates a title after identifying each object present in each data point 11 and assigning a label to each object. The title is a text corpus associated with and describing each identified object in data point 11. According to one embodiment of the invention, an object will have an associated generated title. The control unit 10 derives metadata for each expert model 15 from each expert model 15. For example, for semantic segmentation expert model 15(c), the metadata includes: {car, road, road marker, crosswalk, pedestrian, traffic light, traffic sign, building, curb, sky}. This metadata can be converted into individual phrases / titles, as described below: { Phrases: "Car road near the center" Phras: "Closer to the left-hand driveway" Phrases: "Pedestrian crossing on the right" Phrases: "Pedestrian crossing near the right-hand traffic sign" }
[0012] The multiple titles generated associated with each data point 11 are derived from the exported metadata. The control unit 10 uses any other base model or large language model fusion unit 18 to convert the metadata into titles. A data point 11 is mapped to either a single generated title or multiple generated titles. In one scenario, a data point 11 will have a single generated title, while in another scenario, a data point 11 will have multiple generated titles.
[0013] To fine-tune the intelligent model / base model 20, the generated titles are provided to the text encoder 16, and the data points 11 are provided to the image encoder 14, which uses a contrastive training technique to map the data points 11 to (multiple) related generated titles. It should be noted that the mapping technique described above can be any other technique used to connect / map images / data points 11 to corresponding and related generated title / text corpora. This facilitates efficient performance of complex image searches using the base / intelligent model 20. In addition, the intelligent model 20 generates a parallel corpus of data (obtained by mapping (multiple) generated titles to each data point 11), which encompasses domain-specific information about the data.
[0014] Figure 2 The illustration depicts a method for fine-tuning an intelligent model using a control unit according to the present invention. In step S1, data from multiple sources 12 is received for training and developing an intelligent model 20. In step S2, at least one expert model 15 identifies each object present in each data point 11 of the received data and assigns it a label. In step S3, metadata about each of the at least one expert model 15 is derived, and a large language model fusion processor 18 is used to generate multiple titles. In step S4, the multiple titles generated associated with each data point 11 are sent to a text encoder 16, which maps at least one generated title to a corresponding data point 11. In step S5, after mapping the data points 11 to the corresponding at least one generated title, a contrastive training technique is used to fine-tune the intelligent model 20.
[0015] This method is explained in detail. Base Model 20 (FM) / Intelligent Models (such as CLIP) are jointly trained on a large text data corpus along with visual data (such as images). The scale of the data used to train Base Model 20 is a major driver behind the successful performance of these models 20 on a wide range of unseen downstream tasks. Therefore, these models 20 have great potential to provide solutions for domain-specific use cases in language, vision, robotics, and reasoning tasks. However, text data is inherently very general. To use such models for use cases such as search, retrieval, and filtering, they need to be adapted to data specific to the domain of interest (e.g., autonomous driving data). For example, consider a task of text-to-image search using any of the Base Model 20. Because the data used to train Base / Intelligent Model 20 is not task-specific and contains many general scenarios and titles, while the technique is suitable for example sets, performance gaps exist when it comes to compound queries, verb relations, and smaller scene object details. For example, given the complexity of driving scenarios, a framework needs to be developed for such title-based searches. Furthermore, the same framework can be extended to other fields, such as industrial robots, security and monitoring systems, etc.
[0016] Control unit 10 receives data comprising a large corpus of unlabeled images whose descriptions are constructed in an unsupervised manner using expert model 15. The proposed method also processes individual phrases / headings from each expert model 15 without combining them into a larger heading. The method leverages domain-specific data by generating multiple headings per image using a deep learning model 15 trained on this data. This data can be used to train or fine-tune a visual language base model 20 (e.g., CLIP[1], BLIP[2], etc.). Once trained, this intelligent / base model 20 provides a much more accurate filtering solution even for unseen datasets. The disclosed method can be applied to any other complex scenarios from other domains, such as in-vehicle images, images from household appliances (such as refrigerators and vacuum cleaners), etc.
[0017] Data comprising multiple images 11 is received from multiple sources 12 into a control unit 10 for training and developing an intelligent / base model 20. An expert model 15 present in the control unit 10 identifies each object present in each data point 11 of the data and assigns a label to each identified object.
[0018] The labels provided by expert model 15 are thus fused together / classified using a large language foundation model fusion unit 18 to generate titles. Each data point 11 includes multiple titles associated with the objects / labels present in the image / data point 11. The generated titles are provided to intelligent model 20 via text encoder 16, and simultaneously, multiple images are provided to intelligent model 20 via image encoder 14. During fine-tuning, control unit 10 uses constructive learning techniques to map the associated generated titles to their corresponding images 11.
[0019] The above method is illustrated using an example involving data related to autonomous driving. Any development of the intelligent / base model 20 typically requires acquiring massive amounts of data. Furthermore, from the acquired data, a subset is selected for labeling to train specific perception tasks, such as lane segmentation, person detection, etc. A byproduct of having such labeled data is that large, accurate models can be trained for the same task, and these models can run offline without being limited by memory or runtime. Such models are referred to as "expert models" 15. Consider an example of a specific expert model 15 that anchors (grounds) different objects present in the driving scene image, such as… Figure 1 As shown. One such expert model 15 is the semantic segmentation model 15(c). The semantic segmentation expert model 15(c) provides important information about verb relationships between objects, which can be easily derived. This can take the form of: whether the object is on the sidewalk or on the road, and the direction of the object (left or right). For example, the first line 17(b) on the right indicates the relationship "pedestrian on the sidewalk". Similarly, the second line 17(a) on the left indicates the relationship "car on the road".
[0020] Another expert model is the automatic labeling expert model 15(b). The automatic labeling classifier is used to identify road signs. Expert model 15(b) can involve object attributes. For example, a classifier that can identify road signs is called an "automatic labeling classifier." Another expert model 15(a), involving the entire scene, can be used to identify the weather in a specific image, such as rainy, sunny, or cloudy. Any number of such related expert models (15(a) / 15(b) / 15(c)) are conceivable and adopted in control unit 10. Control unit 10 derives metadata for each expert model 15. Control unit 10 derives metadata from semantic segmentation expert model 15(c), such as: {cars, roads, road markers, crosswalks, pedestrians, traffic lights, traffic signs, buildings, curbs, sky}, and this metadata is converted into individual phrases / headings, as described below: { Phrases: "Car road near the center" Phras: "Closer to the left-hand driveway" Phrases: "Pedestrian crossing on the right" Phrases: "Pedestrian crossing near the right-hand traffic sign" }
[0021] The generated titles are mapped to images 11 present in the data, and the intelligent / base model 20 is fine-tuned to perform complex image searches. Alternatively, it can be captured as more comprehensive titles that incorporate all metadata. In the first case, the control unit 10 obtains tuples (data structures composed of multiple parts) consisting of images 11 and phrase sets, while for the second case, the control unit 10 obtains image and title pairs. Thus, the disclosed method described above generates a parallel corpus (text and image pairs) of data that encompasses domain-specific information about the data, making it more informative and suitable for the initial use case. Using this parallel corpus, the control unit 10 can either train or develop the intelligent / base model 20 from scratch, or fine-tune the visual language base model 20.
[0022] It should be understood that Embodiment 10 described above is merely illustrative and does not limit the scope of the invention. Many such embodiments, as well as other modifications and variations of the embodiments described in the description, are contemplated. The scope of the invention is limited only by the scope of the claims.
Claims
1. A control unit (10) for fine-tuning an intelligent / basic model (20), said control unit (10) being configured to: Data is received from multiple sources (12) for training and developing the intelligent model (20). Each object present in each data point (11) of the received data is identified by at least one expert model (15) and a label is assigned to it; Export metadata about each of the at least one expert model (15), and generate multiple titles using the large language model fusion tool (18); The generated titles associated with each data point (11) are sent to a text encoder (16) which maps at least one of the generated titles to the corresponding data point (11). The intelligent model (20) is fine-tuned using a contrastive training technique after the data points (11) are mapped to at least one of the corresponding generated titles.
2. The control unit (10) as claimed in claim 1, wherein the multiple headers generated in relation to each data point (11) are derived from the exported metadata.
3. The control unit (10) of claim 1, wherein the received data represents a plurality of images (11), and each data point represents an image (11) from the plurality of images (11) received in the group.
4. The control unit (10) as claimed in claim 1, wherein each expert model (15) performs a specific function to provide a label for each object present in each data point / image (11).
5. The control unit (10) as claimed in claim 4, wherein the expert model (15) is selected from the following group of expert models (15): including zero-sample label expert model 15(a), semantic segmentation expert model 15(c), and automatic labeling expert model (15(b)).
6. The control unit (10) of claim 5, wherein each expert model (15) is associated with the attributes of the object in each data point (11), one such expert model (15) – automatic labeling expert model 15(b) – is used to identify all road signs, and another expert model 15(a) is associated with the entire scene to identify the weather in each data point (11).
7. The control unit (10) as claimed in claim 1, wherein one of the data points (11) is mapped to any of the following titles: including a single generated title and multiple generated titles.
8. The control unit (10) of claim 1, wherein the generated title is provided to the text encoder (16) and the data points (11) are provided to the image encoder (14) for mapping the data points (11) to the associated generated title using a contrastive training technique.
9. The control unit (10) of claim 8, wherein the intelligent model is adapted to generate a parallel corpus of data (obtained by mapping one or more generated titles to each data point (11)), the corpus encompassing domain-specific information of the data.
10. A method for fine-tuning an intelligent / basic model (20) by a control unit (10), the method comprising the steps of: Data is received from multiple sources (12) for training and developing the intelligent model (20). Each object present in each data point (11) of the received data is identified by at least one expert model (15) and a label is assigned to it; Export metadata about each of the at least one expert model (15), and generate multiple titles using the large language model fusion tool (18); The generated titles associated with each data point (11) are sent to a text encoder, which maps at least one of the generated titles to the corresponding data point (11). The intelligent model (20) is fine-tuned using a contrastive training technique after the data points (11) are mapped to at least one of the corresponding generated titles.