Label evaluation method and device based on multi-modal large model, medium and vehicle
By using a multimodal large-scale model-based label evaluation method, the output results of the label generation tool are automatically evaluated, solving the problem of low efficiency in manual evaluation in existing technologies. This achieves efficient and accurate evaluation of the label generation tool, and is suitable for large-scale data evaluation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHIJIA MAINLAND (BEIJING) INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-04-24
AI Technical Summary
Existing label generation tools rely on manual sampling for result evaluation, which is inefficient and costly, and is difficult to adapt to rapid evaluation of large amounts of data.
A label evaluation method based on a multimodal large model is adopted. The output results of the label generation tool are automatically evaluated by the label detection model. The recognition results of the label detection model are used as the evaluation benchmark to calculate the accuracy, recall and false detection rate, so as to realize an automated and fast evaluation process.
It improves the efficiency and accuracy of tag generation tool evaluation, reduces labor costs, has high result consistency, is suitable for large-scale data evaluation, avoids human interpretation bias, and supports flexible interaction and rapid acquisition of evaluation reports.
Smart Images
Figure CN121921709A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a label evaluation method, device, medium, and vehicle based on a multimodal large model. Background Technology
[0002] In autonomous driving data processing, label generation tools are typically used to generate scene or behavior labels for a large number of video clips to support data filtering, model training, and scene analysis. Current methods for evaluating the results of label generation tools mainly rely on manual sampling, where individuals manually review selected video clips and compare them with existing labels to estimate the accuracy of the labels.
[0003] However, this method of manually sampling and evaluating the accuracy of the label generation tool's results is inefficient. Summary of the Invention
[0004] The purpose of this application is to provide a label evaluation method, device, medium, and vehicle based on a multimodal large model, which can improve the efficiency of evaluating the accuracy of the results of label generation tools.
[0005] The first aspect of this application provides a label evaluation method based on a multimodal large model. The method includes: receiving an evaluation request for a label generation tool; obtaining a label to be detected, a video to be detected, and a label generation tool according to the evaluation request; performing label recognition on the video to be detected based on the label generation tool, and determining whether the video to be detected includes the label to be detected based on the recognition result; performing label recognition on the video to be detected based on the label detection model, and determining whether the video to be detected includes the label to be detected based on the recognition result; and evaluating the label generation tool based on the difference between the recognition result of the label generation tool and the recognition result of the label detection model.
[0006] The evaluation method for the aforementioned tag generation tool obtains accurate tag detection results based on the tag detection model, and evaluates the tag generation tool based on the tag detection results of the tag detection model. This avoids manual frame-by-frame inspection, has a high degree of automation, reduces the labor cost of evaluation, and improves the efficiency of evaluating the accuracy of the tag generation tool and its output results.
[0007] Furthermore, this method can directly and quickly evaluate large batches of data to be tested, is easy to scale, and the evaluation process is based on a unified model and rules, avoiding the difference in results caused by human interpretation bias, resulting in high consistency of results.
[0008] In one possible implementation of the first aspect above, obtaining the tag to be detected, the video to be detected, and the tag generation tool according to the evaluation request includes: parsing the evaluation request to obtain the tag to be detected and the video to be detected; and obtaining the tag generation tool corresponding to the tag to be detected from the detection database according to the tag to be detected.
[0009] In one possible implementation of the first aspect described above, the identification result of the tag generation tool includes at least one video segment and indication information on whether the at least one video segment has a tag to be detected.
[0010] In one possible implementation of the first aspect described above, the identification result of the label detection model includes at least one video segment and indication information on whether at least one video segment has a label to be detected.
[0011] In one possible implementation of the first aspect above, label recognition of the video to be detected is performed based on the label detection model, including: inputting the video to be detected and the label to be detected into the label detection model, and performing label recognition.
[0012] In one possible implementation of the first aspect above, tag recognition of the video to be detected is performed based on the tag detection model, including: inputting at least one video segment and the tag to be detected from the recognition result of the tag generation tool into the tag detection model to perform tag recognition.
[0013] In one possible implementation of the first aspect above, tag recognition of the video to be detected based on the tag detection model includes: determining a video segment containing a tag to be detected based on indication information corresponding to at least one video segment in the recognition result of the tag generation tool; and inputting the video segment containing the tag to be detected and the tag to be detected into the tag detection model for tag recognition.
[0014] In one possible implementation of the first aspect above, tag recognition of the video to be detected based on the tag detection model includes: determining video segments that do not include the tag to be detected based on the indication information corresponding to at least one video segment in the recognition result of the tag generation tool; and inputting the video segments that do not include the tag to be detected and the tag to be detected into the tag detection model to perform tag recognition.
[0015] In one possible implementation of the first aspect above, the tag generation tool is evaluated based on the difference between the recognition results of the tag generation tool and the recognition results of the tag detection model, including: using the recognition results of the tag detection model as the evaluation benchmark, and obtaining the accuracy of the tag generation tool based on the difference between the recognition results of the tag detection model and the recognition results of the tag generation tool; and / or, using the recognition results of the tag detection model as the evaluation benchmark, and obtaining the recall of the tag generation tool based on the difference between the recognition results of the tag detection model and the recognition results of the tag generation tool; and / or, using the recognition results of the tag detection model as the evaluation benchmark, and obtaining the video segments that are incorrectly judged in the recognition results of the tag detection model based on the difference between the recognition results of the tag detection model and the recognition results of the tag generation tool.
[0016] In one possible implementation of the first aspect above, the data type of the data to be detected in the evaluation request includes at least one of video data, audio data, text data, and image data, and the evaluation request is used to indicate the label to be detected and the data to be detected corresponding to the label.
[0017] In one possible implementation of the first aspect described above, the label detection model includes a visual language model.
[0018] The second aspect of this application provides an electronic device, including a processor and an interface circuit. The interface circuit is used to receive signals from other communication devices and transmit them to the processor, or to send signals from the processor to other communication devices. The processor implements the tag evaluation method based on a multimodal large model as described in the first aspect and any implementation thereof through logic circuits or executing code instructions.
[0019] A third aspect of this application provides a computer-readable storage medium storing a computer program or instructions that, when executed by a communication device, implement the label evaluation method based on a multimodal large model as described in the first aspect and any implementation thereof.
[0020] The fourth aspect of this application provides a program product including instructions that, when executed, enable the implementation of a label evaluation method based on a multimodal large model as described in the first aspect and any implementation thereof.
[0021] A fourth aspect of this application provides a vehicle that includes the electronic equipment mentioned in the second aspect. Attached Figure Description
[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0023] Figure 1 A flowchart illustrating a label evaluation method based on a multimodal large model is shown according to an embodiment of this application;
[0024] Figure 2 A schematic diagram of a module of an electronic device 200 is shown according to an embodiment of this application;
[0025] Figure 3 A schematic diagram of the hardware structure of a vehicle 100 is shown according to an embodiment of this application. Detailed Implementation
[0026] The illustrative embodiments of this application include, but are not limited to, a label evaluation method, apparatus, medium, and vehicle based on a multimodal large model.
[0027] As mentioned earlier, in autonomous driving data processing, existing label evaluation methods mainly rely on manual assessment of the accuracy of the output results of label generation tools. For example, the manual evaluation process may include: manually extracting a certain number of representative segments from a large batch of autonomous driving video clips, manually reviewing and comparing the output results of the label generation tool with the labels segment by segment, recording the correctness and errors of the output results, and finally manually calculating the accuracy indicators and compiling the evaluation conclusions. Understandably, this method of manually sampling and evaluating the accuracy of the label generation tool's output results is inefficient.
[0028] In view of this, embodiments of this application provide a label evaluation method based on a multimodal large model. This method can receive an evaluation request for a label generation tool; based on the evaluation request, obtain the label to be detected, the video to be detected, and the label generation tool; perform label recognition on the video to be detected based on the label generation tool, and determine whether the video to be detected includes the label to be detected based on the recognition result; perform label recognition on the video to be detected based on the label detection model, and determine whether the video to be detected includes the label to be detected based on the recognition result; and evaluate the label generation tool based on the difference between the recognition result of the label generation tool and the recognition result of the label detection model.
[0029] This method obtains accurate label detection results based on a label detection model, and evaluates the label generation tool based on the label detection results of the label detection model. This avoids manual frame-by-frame inspection, has a high degree of automation, reduces the labor cost of evaluation, and improves the efficiency of evaluating the accuracy of the label generation tool and its output results.
[0030] Furthermore, this method can directly and quickly evaluate large batches of data to be tested, is easy to scale, and the evaluation process is based on a unified model and rules, avoiding the difference in results caused by human interpretation bias, resulting in high consistency of results.
[0031] The method provided in this application can be applied to any electronic device, including but not limited to mobile stations (MS) and mobile terminals (MT). For example, the electronic device can be an in-vehicle device, a mobile phone, a smart TV, a wearable device, a tablet computer, a virtual reality (VR) device, an augmented reality (AR) device, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical surgery, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, and so on. This application does not limit the specific form of the electronic device.
[0032] In the embodiments of this application, "electronic device" can refer to the electronic device itself, or a component in the electronic device (e.g., a processor, chip, or chip system), or a logic module or software that can realize all or part of the functions of the electronic device.
[0033] Figure 1 A schematic flowchart of a label evaluation method based on a multimodal large model is shown according to an embodiment of this application. It can be understood that... Figure 1 The execution entities for each step of the process shown are electronic devices. For simplicity, the following description... Figure 1 The execution entity for each step in the illustrated process will not be described again. For example... Figure 1 As shown, this process includes, but is not limited to, the following:
[0034] S110, Received a request to evaluate the tag generation tool.
[0035] In some embodiments, users can input evaluation requests for the tag generation tool into an electronic device in various ways. For example, a user can input a database index into the electronic device, allowing the device to retrieve the evaluation request from the database based on the index. Alternatively, a user can upload the evaluation request to the electronic device. The format of the evaluation request may include at least one of video, audio, text, and motion. In this embodiment, no specific limitations are made on the method or format of the user inputting the evaluation request.
[0036] In some embodiments, an evaluation request is used to indicate the label to be detected and the corresponding data to be detected. The evaluation request may include evaluating the output obtained by the label generation tool based on the data to be detected and the label to be detected.
[0037] For example, an evaluation request may include evaluating the output of the label generation tool based on the data to be detected and the nearby large vehicle (i.e., the label to be detected). In some embodiments, the data type of the data to be detected may include at least one of video data, audio data, text data, and image data. In this application embodiment, the data type of the data to be detected is not specifically limited.
[0038] For ease of description, the embodiments of this application use the video to be detected as an example of the data to be detected, and a large vehicle at close range as an example of the tag to be detected.
[0039] S120, based on the evaluation request, obtains the tags to be tested, the videos to be tested, and the tag generation tool.
[0040] In some embodiments, the target tag and the target video can be obtained by parsing the evaluation request, and then the tag generation tool corresponding to the target tag can be retrieved from the detection database based on the target tag. It is understood that the detection database can store tag generation tools corresponding to various tags, as well as the correspondence between various tags and tag generation tools. Thus, the tag generation tool corresponding to the target tag can be retrieved from the database based on the target tag and the correspondence between various tags and tag generation tools. The detection database can be any type of database; no specific limitation is made in this embodiment.
[0041] For example, a user can input an evaluation request via natural language: "Help me evaluate the prediction accuracy of the tag generation tool for the tag 'near-distance large vehicle' using [video clip 1, video clip 2]". The evaluation request can be parsed to obtain the videos to be detected as video clip 1 and video clip 2, and the tag to be detected as "near-distance large vehicle". Then, based on "near-distance large vehicle", the corresponding tag generation tool can be obtained from the detection database.
[0042] S130: The tag generation tool performs tag recognition on the video to be detected, and determines whether the video to be detected includes the tag to be detected based on the recognition result.
[0043] In some embodiments, the recognition result of the tag generation tool may include at least one video segment and indication information indicating whether the at least one video segment has the tag to be detected. The indication information may be in text format, voice format, video format, logical format (0 or 1), etc. No specific limitation is made in this embodiment. In some embodiments, the video to be detected and the tag to be detected may be input into the tag generation tool for tag recognition, and the recognition result of the tag generation tool may be obtained. Based on the recognition result, it may be determined whether the video to be detected includes the tag to be detected.
[0044] For example, when the video to be detected is the aforementioned video segment 1 and video segment 2, such as video segment 1 being a video segment of 0-10 seconds and video segment 2 being a video segment of 11-12 seconds, the recognition result of the tag generation tool may include:
[0045] Video clip A (0-5s): contains close-up shots of a large vehicle;
[0046] Video clip B (8-10s): contains close-up views of a large vehicle; and / or,
[0047] Video clip C (6-7s): does not include large vehicles at close range;
[0048] Video clip D (11-12s): does not include large vehicles at close range.
[0049] In this video segment, video clips A, B, C, and D are all segments from the video to be detected, with "near-distance large vehicle" as the label to be detected. The indication information for video clip A is "Contains near-distance large vehicle," indicating that video clip A has the label to be detected, i.e., "near-distance large vehicle." The indication information for video clip B is "Contains near-distance large vehicle," indicating that video clip B has the label to be detected, i.e., "near-distance large vehicle." The indication information for video clip C is "Does not contain near-distance large vehicle," indicating that video clip C does not have the label to be detected, i.e., it does not contain "near-distance large vehicle." The indication information for video clip D is "Does not contain near-distance large vehicle," indicating that video clip D does not have the label to be detected, i.e., it does not contain "near-distance large vehicle."
[0050] It should be noted that video clip A (0-5s) refers to the video clip from 0-5s in video clip 1. Video clip B (8-10s) refers to the video clip from 8-10s in video clip 1. Video clip C (6-7s) refers to the video clip from 6-7s in video clip 1. Video clip D (11-12s) refers to the video clip from 11-12s in video clip 2, and so on.
[0051] S140, Based on the label detection model, perform label recognition on the video to be detected, and determine whether the video to be detected includes the label to be detected based on the recognition result.
[0052] In some embodiments, the label detection model may include a visual language model, such as a multimodal visual language model or other large multimodal model. In this application embodiment, no specific limitation is made to the label detection model.
[0053] In some embodiments, the identification results of the label detection model include at least one video segment and indication information on whether at least one video segment has the label to be detected.
[0054] In some embodiments, tag recognition of the video to be detected based on the tag detection model includes: inputting at least one video segment and the tag to be detected from the recognition result of the tag generation tool into the tag detection model to perform tag recognition.
[0055] For example, the video clips A, B, C, and D from the tag generation tool's recognition results, along with the tag to be detected, can be input into the tag detection model for tag recognition.
[0056] The recognition results of the label detection model include:
[0057] Does video clip A (0-5s) contain a large vehicle at close range? Yes;
[0058] Does video clip B (8-10s) contain a large vehicle at close range? No.
[0059] Does video clip C (6-7s) contain a large vehicle at close range? Yes;
[0060] Does video clip D (11-12s) contain a large vehicle at close range? No.
[0061] Among them, video clips A, B, C, and D are video clips in the video to be detected, and the large vehicle at close range is the label to be detected. The recognition result of the label detection model indicates that the label detection model confirms that video clip A contains a large vehicle at close range, video clip B does not contain a large vehicle at close range, video clip C contains a large vehicle at close range, and video clip D does not contain a large vehicle at close range.
[0062] In some embodiments, tag recognition of the video to be detected based on the tag detection model includes: determining video segments including tags to be detected based on indication information corresponding to at least one video segment in the recognition results of the tag generation tool; and inputting the video segments including the tags to be detected and the tags to be detected into the tag detection model for tag recognition.
[0063] For example, if the identification results of the tag generation tool determine that video segment A and video segment B contain the tag to be detected based on the indication information of the aforementioned video segment A and video segment B, then video segment A containing the tag to be detected, video segment B containing the tag to be detected, and the tag to be detected can be input into the tag detection model for tag recognition.
[0064] The recognition results of the label detection model include:
[0065] Does video clip A (0-5s) contain a large vehicle at close range? Yes;
[0066] Does video clip B (8-10s) contain a large vehicle at close range? No.
[0067] In this context, video clip A and video clip B are video clips from the video to be detected, and the large vehicle at close range is the label to be detected. The recognition result of the label detection model indicates that the label detection model confirms that video clip A contains a large vehicle at close range, while video clip B does not contain a large vehicle at close range.
[0068] In some embodiments, tag recognition of the video to be detected based on the tag detection model further includes: determining video segments that do not include the tag to be detected based on indication information corresponding to at least one video segment in the recognition results of the tag generation tool; and inputting the video segments that do not include the tag to be detected and the tag to be detected into the tag detection model for tag recognition.
[0069] For example, if the identification results of the tag generation tool determine that video segment C and video segment D do not include the tag to be detected based on the indication information of the video segment C and video segment D mentioned above, then the video segment C without the tag to be detected, the video segment D without the tag to be detected, and the tag to be detected can be input into the tag detection model.
[0070] The recognition results of the label detection model include:
[0071] Does video clip C (6-7s) contain a large vehicle at close range? No;
[0072] Does video clip D (11-12s) contain a large vehicle at close range? No.
[0073] Among them, video clip C and video clip D are video clips in the video to be detected, and the large vehicle at close range is the label to be detected. The recognition result of the label detection model indicates that the label detection model confirms that video clip C contains a large vehicle at close range, while video clip D does not contain a large vehicle at close range.
[0074] In some embodiments, label recognition of the video to be detected based on the label detection model includes: inputting the video to be detected and the label to be detected into the label detection model and performing label recognition.
[0075] For example, the recognition results of the label detection model include:
[0076] Does video clip A (0-5s) contain a large vehicle at close range? Yes;
[0077] Does video clip B (8-10s) contain a large vehicle at close range: No; and / or,
[0078] Does video clip C (6-7s) contain a large vehicle at close range? Yes;
[0079] Does video clip D (11-12s) contain a large vehicle at close range? No.
[0080] Among them, video clips A, B, C, and D are segments in the video to be detected, and the large vehicle at close range is the label to be detected. The recognition result of the label detection model indicates that the label detection model confirms that video clip A contains a large vehicle at close range, video clip B does not contain a large vehicle at close range, video clip C contains a large vehicle at close range, and video clip D does not contain a large vehicle at close range.
[0081] Understandably, on the one hand, inputting the video to be detected and the tag to be detected into the tag detection model allows for the acquisition of complete and accurate recognition results from the tag detection model using the complete input video. On the other hand, inputting video segments whose recognition results from the tag generation tool include the tag to be detected, and / or video segments whose recognition results from the tag generation tool do not include the tag to be detected, along with the tag itself, into the tag detection model can reduce the computation time of the tag detection model and further improve the efficiency of evaluating the accuracy of the tag generation tool and its output.
[0082] S150. The label generation tool is evaluated based on the difference between the recognition results of the label generation tool and the recognition results of the label detection model.
[0083] In some embodiments, the recognition result of the label detection model can be used as the evaluation benchmark (i.e., the correct evaluation result). Based on the difference between the recognition result of the label detection model and the recognition result of the label generation tool, the accuracy of the label generation tool and its recognition result can be evaluated to obtain the accuracy rate of the label generation tool.
[0084] Table 1 shows the recognition results of the label generation tool and the label detection model.
[0085] Table 1:
[0086]
[0087] Therefore, referring to Table 1, the tag generation tool correctly identified video clips A and D, and incorrectly identified video clips B and C. Thus, the total number of video clips is 4, the number of correctly identified video clips is 2, and the accuracy rate of the tag generation tool is 50%.
[0088] In some embodiments, the recognition results of the label detection model can be used as the evaluation benchmark. The recall rate of the label generation tool is obtained based on the difference between the recognition results of the label detection model and the recognition results of the label generation tool. Referring to Table 1, the label generation tool correctly identified video segments A and D, but incorrectly identified video segments B and C. Therefore, the total number of video segments is 4, the number of incorrectly identified video segments is 2, and the recall rate of the label generation tool is 50%.
[0089] In some embodiments, the recognition results of the label detection model can be used as an evaluation benchmark. Based on the difference between the recognition results of the label detection model and the recognition results of the label generation tool, video segments in which the recognition results of the label generation tool are incorrectly judged can be obtained.
[0090] For example, referring to Table 1, for video segment A (8-10s), the label generation tool's identification result is that it includes the label to be detected, while the label detection model's identification result is that it does not include the label to be detected; while for video segment C (6-7s), the label generation tool's identification result is that it does not include the label to be detected, while the label detection model's identification result is that it includes the label to be detected. Therefore, video segments A (8-10s) and C (6-7s) are falsely detected video segments, that is, video segments that are judged incorrectly.
[0091] In some embodiments, the recognition results of the label detection model can be used as an evaluation benchmark. Based on the difference between the recognition results of the label detection model and the recognition results of the label generation tool, the video segments missed by the label generation tool can be obtained. For example, if the label recognition results of the label generation tool include video segment A (0-5s): containing a large vehicle at close range;
[0092] Video clip B (8-10s): contains a large vehicle at close range; however, the label detection model's identification result includes: Does video clip E (13-15s) contain a large vehicle at close range? Yes, therefore the label generation tool's identification result does not include video clip E (13-15s). Video clip E (13-15s) is a video clip that the label generation tool missed.
[0093] The aforementioned label evaluation method based on a multimodal large model can obtain the accuracy of the label generation tool by comparing the results of two types of labels, thus improving the efficiency of accuracy evaluation of the label generation tool and its output. This method is highly automated, eliminating the need for manual frame-by-frame checking, reducing labor costs, and is easily scalable. It can directly and quickly perform automatic evaluation on large batches of data to be tested, making it suitable for large-scale data systems. Furthermore, the results are highly consistent, as the evaluation process is based on a unified model and rules, avoiding discrepancies caused by human interpretation bias. Simultaneously, this method supports flexible interaction, allowing users to interact with electronic devices through flexible input methods to quickly obtain evaluation details or reports, improving usability.
[0094] It is understood that in other embodiments, depending on actual needs, the steps shown in the above embodiments can be combined, deleted, or replaced with other steps that are beneficial to achieving the purpose of this application, and this application does not impose any restrictions here.
[0095] This application also provides a tag evaluation system based on a multimodal large model. The evaluation system for the tag generation tool may include at least one processor, which executes instructions stored in a non-transitory computer-readable medium such as memory. The processor may be any conventional processor, such as a commercially available central processing unit (CPU). Optionally, the processor may be a dedicated device such as an application-specific integrated circuit (ASIC) or other hardware-based processor.
[0096] In some embodiments, the memory may contain instructions (e.g., program logic) that can be executed by a processor to perform various functions of the electronic device, including the functions of the embodiments described above.
[0097] This application also provides an electronic device 200, which may include at least one processor that executes instructions stored in a non-transitory computer-readable medium such as memory. The processor may be any conventional processor, such as a commercially available CPU. Optionally, the processor may be a special-purpose device such as an ASIC or other hardware-based processor. The memory may contain instructions (e.g., program logic) that can be executed by the processor to perform various functions of the vehicle 100, including the functions of the embodiments described above.
[0098] Next, combine Figure 2 The structure of electronic device 200 will be described.
[0099] like Figure 2 As shown, the electronic device 200 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output device 105, and system control logic unit 106 for coupling the processor 101, system memory 102, non-volatile memory 103, communication interface 104, and input / output (I / O) device 105. Wherein:
[0100] Processor 101 may include one or more processing units, such as a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), microprocessor (MCU), artificial intelligence (AI) processor, field programmable gate array (FPGA), neural network processing unit (NPU), etc., or a processing module or processing circuit that may include one or more single-core or multi-core processors. In some embodiments, the CPU may be used to optimize the neural network model to be run, and the NPU may be used to run the neural network model to be run. Processor 101 is used to execute any of the above-described label evaluation methods based on a multimodal large model.
[0101] System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions. For example, in some embodiments, system memory 102 can be used to store data provided by the aforementioned different services, such as sensor data, image data, or video data, and can also be used to store instructions for the label evaluation method based on the multimodal large model provided in the aforementioned embodiments.
[0102] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), compact disc (CD), digital versatile disc (DVD), solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital (SD) memory card. In other embodiments, the non-volatile memory 103 may be used to store instructions, etc., of the tag evaluation method based on multimodal large models provided in the foregoing embodiments.
[0103] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by at least one of processors 101, causing electronic device 200 to implement the tag evaluation method based on multimodal large model provided in the embodiments of this application.
[0104] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the electronic device 200, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the electronic device 200, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the electronic device 200 may communicate with other devices through the communication interface 104, for example, the electronic device 200 may obtain relevant data from other devices through the communication interface 104.
[0105] Input / output (I / O) device 105 can be an input device such as a keyboard or mouse, and an output device such as a monitor. Users can interact with electronic device 200 through input / output (I / O) device 105.
[0106] The system control logic unit 106 may include any suitable interface controller to provide any suitable interface to other modules of the electronic device 200. For example, in some embodiments, the system control logic unit 106 may include one or more memory controllers to provide an interface to the system memory 102 and the non-volatile memory 103.
[0107] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for the system control logic unit 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for the system control logic unit 106 to form a system-on-chip (SoC).
[0108] Understandable. Figure 2 The structure of the electronic device 200 shown is merely an example. In other embodiments, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0109] This application also provides a vehicle, which may include electronic devices, such as vehicle-mounted systems or in-vehicle equipment.
[0110] Next, combine Figure 3 The structure of the vehicle will be described. Figure 3 A schematic diagram of a vehicle structure provided in an embodiment of this application is shown.
[0111] Figure 3 This is a schematic diagram illustrating a possible functional framework of a vehicle 100 provided in an embodiment of this application. For example... Figure 3As shown, the functional framework of vehicle 100 may include various subsystems, such as the sensor system 10, control system 20, one or more peripheral devices 30 (one is shown as an example), power supply 40, and on-board equipment 50. Optionally, vehicle 100 may also include other functional systems, such as an engine system that provides power to vehicle 100, etc., which are not limited here. The sensor system 10 may include several detection devices that can sense the measured information and convert the sensed information into electrical signals or other required forms of information output according to a certain rule. As shown, these detection devices may include a global positioning system 11 (GPS), a vehicle speed sensor 12, an inertial measurement unit 13 (IMU), etc., which are not limited here. The global positioning system 11 is a system that uses GPS positioning satellites to perform real-time positioning and navigation globally. In this application, the global positioning system 11 can be used to realize the real-time positioning of vehicle 100 and provide the geographical location information of vehicle 100. The vehicle speed sensor 12 is used to detect the driving speed of vehicle 100. The inertial measurement unit 13 may include a combination of an accelerometer and a gyroscope, and is a device for measuring the angular rate and acceleration of the vehicle 100. For example, during the movement of the vehicle 100, the inertial measurement unit can measure the changes in the position and angle of the vehicle body based on the inertial acceleration of the vehicle 100, such as measuring the acceleration and angular rate of the vehicle 100. The control system 20 may include a steering unit 21, a braking unit 22, etc. The steering unit 21 may represent a system for adjusting the direction of travel of the vehicle 100, which may include, but is not limited to, a steering wheel, or any other structural device for adjusting or controlling the direction of travel of the vehicle 100. The braking unit 22 may represent a system for slowing down the speed of the vehicle 100, and may also be called the vehicle 100 braking system. It may include, but is not limited to, a brake controller, a reducer, or any other structural device for slowing down the vehicle 100. In practical applications, the braking unit 22 may use friction to slow down the tires of the vehicle 100, thereby slowing down the speed of the vehicle 100. The peripheral equipment 30 may include several components, such as the communication system 31, touch screen 32, user interface 33, etc. shown in the figure. The communication system 31 is used to enable network communication between the vehicle 100 and other devices besides the vehicle 100, such as the in-vehicle device 50. In practical applications, the communication system 31 can use wireless communication technology or wired communication technology to enable network communication between the vehicle 100 and other devices. The wired communication technology can refer to communication between the vehicle 100 and other devices via network cables or optical fibers.The wireless communication technologies include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), and Time Division Code Division Multiple Access (TDMA). division code division multiple access, TD SCDMA, Long Term Evolution (LTE), Wireless Local Area Networks (WLAN) (such as Wireless Fidelity, Wi-Fi) The system includes technologies such as Fi (Fi) networks, Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR). The touchscreen 32 can be used to detect operation commands displayed on it. For example, a user can perform touch operations on the content displayed on the touchscreen 32 according to actual needs to achieve the corresponding function, such as playing music, video, or other multimedia files. The user interface 33 can specifically be a touch panel, used to detect operation commands on it. The user interface 33 can also be a physical button or a mouse. The user interface 33 can also be a display screen, used to output data and display images or data. Optionally, the user interface 33 can also be at least one device belonging to the category of peripheral devices, such as a touchscreen, microphone, and speaker. Several functions of the vehicle 100 are controlled and implemented by the on-board equipment 50. The on-board equipment 50 may include multiple processors such as a general-purpose processor 51, CDC 52, MDC 53, T... BOX 54, as well as memory 55 (also known as storage device) and gateway 56.
[0112] In practical applications, the memory 55 can be located either inside or outside the vehicle-mounted device 50, such as as a cache in the vehicle 100; this application does not limit its location. The general-purpose processor 51 can be, for example, a graphics processing unit (GPU). General-purpose processor 51, CDC 52, MDC 53, T BOX 54 can be used to run relevant programs or corresponding instructions stored in memory 55 to implement the corresponding functions of vehicle 100, such as network switching functions based on services. Memory 55 may include volatile memory, such as RAM; memory may also include non-volatile memory. Memory 55 may include volatile memory, such as ROM, flash memory, HDD, or SSD; memory 55 may also include combinations of the above types of memory. Memory 55 can be used to store a set of program code or instructions corresponding to program code, so that the general-purpose processor 51 can call the program code or instructions stored in memory 55 to implement the corresponding functions of vehicle 100. This function includes, but is not limited to, […]. Figure 3 The schematic diagram of the functional framework of vehicle 100 shown illustrates some or all of the functions. In this application, memory 55 can store a set of program code for controlling vehicle 100, including general-purpose processor 51, CDC 52, MDC 53, and T... The BOX 54 can call this program code to control vehicle 100 to perform the vehicle network switching in this application.
[0113] Optionally, in addition to storing program code or instructions, memory 55 may also store road maps, driving lines, or other structural devices used to adjust or control the direction of travel of vehicle 100. Braking unit 22 may represent a system for slowing down the vehicle 100, or may be referred to as the vehicle 100 braking system. It may include, but is not limited to, a brake controller, a reducer, or other structural devices used for slowing down vehicle 100. In practical applications, braking unit 22 may utilize friction to slow down the tires of vehicle 100, thereby reducing the vehicle 100's speed.
[0114] Peripheral device 30 may include several components, such as the communication system 31, touch screen 32, user interface 33, etc., as shown in the figure. The communication system 31 is used to realize network communication between vehicle 100 and other devices besides vehicle 100, such as in-vehicle equipment 50. In practical applications, the communication system 31 can use wireless communication technology or wired communication technology to realize network communication between vehicle 100 and other devices. The wired communication technology can refer to communication between vehicle 100 and other devices via network cable or fiber optic cable. The wireless communication technology includes, but is not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), and Time Division Code Division Multiple Access (TDMA). division code divisionmultiple access, TD SCDMA, Long Term Evolution (LTE), Wireless Local Area Networks (WLAN) (such as Wireless Fidelity, Wi-Fi) This includes technologies such as Fi (Fi) networks, Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR).
[0115] The touchscreen 32 can be used to detect operation commands on the touchscreen 32. For example, the user can perform touch operations on the content data displayed on the touchscreen 32 according to actual needs to achieve the corresponding function, such as playing music, video, or other multimedia files. The user interface 33 can specifically be a touch panel, used to detect operation commands on the touch panel. The user interface 33 can also be a physical button or a mouse. The user interface 33 can also be a display screen, used to output data and display images or data. Optionally, the user interface 33 can also be at least one device belonging to the category of peripheral devices, such as a touchscreen, microphone, and speaker.
[0116] Several functions of vehicle 100 are controlled and implemented by on-board equipment 50. On-board equipment 50 may include multiple processors such as general-purpose processor 51, CDC 52, MDC 53, T... The system includes a box 54, a memory 55 (also referred to as a storage device), and a gateway 56. In practical applications, the memory 55 can be located either inside or outside the vehicle-mounted device 50, such as as a cache within the vehicle 100; this application does not limit its location. The general-purpose processor 51 can be, for example, a graphics processing unit (GPU). General-purpose processor 51, CDC 52, MDC 53, T... BOX 54 can be used to run relevant programs or instructions stored in memory 55 to implement corresponding functions of vehicle 100, such as network switching function based on services.
[0117] Memory 55 may include volatile memory, such as RAM; memory may also include non-volatile memory. Memory 55 may include volatile memory, such as ROM, flash memory, HDD, or SSD; memory 55 may also include combinations of the above types of memory. Memory 55 can be used to store a set of program code or instructions corresponding to program code, so that the general-purpose processor 51 can call the program code or instructions stored in memory 55 to implement the corresponding functions of vehicle 100. This function includes, but is not limited to, […]. Figure 3 The schematic diagram of the functional framework of vehicle 100 shown illustrates some or all of the functions. In this application, memory 55 can store a set of program code for controlling vehicle 100, including general-purpose processor 51, CDC 52, MDC 53, and T... The BOX 54 can call this program code to control vehicle 100 to perform the vehicle network switching in this application.
[0118] Optionally, in addition to storing program code or instructions, the memory 55 may also store information such as road maps, driving routes, and sensor data. The on-board device 50 can be combined with other components in the functional framework diagram of the vehicle 100, such as sensors in the sensor system and GPS, to realize the relevant functions of the vehicle 100. For example, the on-board device 50 can control the driving direction or speed of the vehicle 100 based on the data input from the sensor system 10; this application does not impose limitations on this.
[0119] It should be noted that the above Figure 2This is merely a schematic diagram of one possible functional framework for vehicle 100. In practical applications, vehicle 100 may include more or fewer systems or components, and this application is not limiting. Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.
[0120] It should be understood that Figure 2 The structure of the vehicle 100 shown is only an example. In other embodiments, the vehicle 100 may include more or fewer modules, which is not limited herein.
[0121] In some embodiments, this application also provides a computer-readable storage medium storing at least one computer program instruction, at least one program segment, code set, or instruction set, which is loaded and executed by a model training system to implement the label evaluation method based on a multimodal large model provided in the above-described method embodiments.
[0122] In some embodiments, this application also provides a computer program product, which includes computer program instructions that, when executed by a model training system, enable the device to implement the label evaluation method based on a multimodal large model provided in the above-described method embodiments.
[0123] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.
[0124] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.
[0125] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.
[0126] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, computer-readable media may include: USB flash drives, external hard drives, read-only memory (ROMs), etc. Various media that can store program code, such as ROM (ROM only memory), RAM (random access memory), magnetic disks, or optical disks.
[0127] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.
[0128] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.
[0129] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0130] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.
Claims
1. A label evaluation method based on a multimodal large model, applied to electronic devices, characterized in that, The method includes: A request to evaluate the tag generation tool has been received. Based on the evaluation request, obtain the tags to be detected, the video to be detected, and the tag generation tool; The video to be detected is identified using the tag generation tool, and the video to be detected is determined to include the tag to be detected based on the identification result. The video to be detected is labeled based on a label detection model, and the video to be detected is determined to include the label to be detected based on the recognition result. The label generation tool is evaluated based on the difference between the recognition results of the label generation tool and the recognition results of the label detection model.
2. The label evaluation method based on a multimodal large model according to claim 1, characterized in that, The step of obtaining the tags to be detected, the video to be detected, and the tag generation tool according to the evaluation request includes: Parse the evaluation request to obtain the tags and videos to be detected; Based on the label to be detected, obtain the label generation tool corresponding to the label to be detected from the detection database.
3. The label evaluation method based on a multimodal large model according to claim 1, characterized in that, The identification result of the tag generation tool includes at least one video segment and indication information on whether the at least one video segment has the tag to be detected.
4. The label evaluation method based on a multimodal large model according to claim 3, characterized in that, The identification result of the tag detection model includes at least one video segment and indication information on whether the at least one video segment has the tag to be detected.
5. The label evaluation method based on a multimodal large model according to claim 3, characterized in that, The tag recognition of the video to be detected based on the tag detection model includes: The video to be detected and the tag to be detected are input into the tag detection model for tag recognition.
6. The label evaluation method based on a multimodal large model according to claim 3, characterized in that, The tag recognition of the video to be detected based on the tag detection model includes: The at least one video segment and the tag to be detected from the recognition results of the tag generation tool are input into the tag detection model for tag recognition.
7. The label evaluation method based on a multimodal large model according to claim 3, characterized in that, The tag recognition of the video to be detected based on the tag detection model includes: Based on the indication information corresponding to at least one video segment in the recognition result of the tag generation tool, the video segment including the tag to be detected is determined; The video clip containing the tag to be detected and the tag to be detected are input into the tag detection model for tag recognition.
8. The label evaluation method based on a multimodal large model according to claim 3, characterized in that, The tag recognition of the video to be detected based on the tag detection model includes: Based on the indication information corresponding to at least one video segment in the recognition results of the tag generation tool, video segments that do not include the tag to be detected are determined. The video clip excluding the tag to be detected and the tag to be detected are input into the tag detection model for tag recognition.
9. The label evaluation method based on a multimodal large model according to claim 1, characterized in that, The evaluation of the tag generation tool based on the difference between the recognition results of the tag generation tool and the recognition results of the tag detection model includes: Using the recognition result of the label detection model as the evaluation benchmark, the accuracy of the label generation tool is obtained based on the difference between the recognition result of the label detection model and the recognition result of the label generation tool; and / or, Using the recognition results of the label detection model as the evaluation benchmark, the recall rate corresponding to the label generation tool is obtained based on the difference between the recognition results of the label detection model and the recognition results of the label generation tool; and / or, The recognition results of the tag detection model are used as the evaluation benchmark. Based on the difference between the recognition results of the tag detection model and the recognition results of the tag generation tool, the video segments that are incorrectly judged in the recognition results of the tag detection model are obtained.
10. The label evaluation method based on a multimodal large model according to claim 1, characterized in that, The data type of the data to be detected in the evaluation request includes at least one of video data, audio data, text data, and image data. The evaluation request is used to indicate the label to be detected and the data to be detected corresponding to the label.
11. The label evaluation method based on a multimodal large model according to claim 1, characterized in that, The label detection model includes a visual language model.
12. An electronic device, characterized in that, include: Memory is used to store instructions executed by one or more processors of an electronic device; And a processor, one of the processors of the electronic device, for executing instructions stored in the memory to implement the label evaluation method based on a multimodal large model as described in any one of claims 1 to 11.
13. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a communication device, implement the label evaluation method based on a multimodal large model as described in any one of claims 1 to 11.
14. A program product, characterized in that, The program product includes instructions that, when executed, enable the label evaluation method based on a multimodal large model as described in any one of claims 1 to 11 to be implemented.
15. A vehicle, characterized in that, Includes the electronic device as described in claim 12.