Data acquisition method and related device thereof

Abstract

Disclosed in the present application are a data acquisition method and a related device thereof, which can greatly reduce human participation during a data acquisition process, thereby saving time cost and labor cost effectively. The method of the present application comprises: first, a terminal device can send to a network device a first image belonging to a first scene; then, the network device can process the first image to obtain a first feature, and send the first feature to the terminal device; subsequently, the terminal device can process a second image to obtain a second feature, and, on the basis of the first feature and the second feature, detect whether the first scene and a second scene are scenes of a same category, and if so, the terminal device sends the second image to the network device; and finally, the network device can use the second image and the first image for subsequent model training.

Description

A data acquisition method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 14, 2023, with application number 202311024088.6 and application name “A data acquisition method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to artificial intelligence (AI) technology, and in particular to a data acquisition method and related equipment. Background Art

[0003] In the autonomous driving field, to continuously optimize autonomous driving models, autonomous driving manufacturers need to deploy dedicated training databases. Acquiring and mining difficult data is a key step in deploying these training databases. With the accelerated rollout of autonomous driving technology, the ability to automatically and efficiently collect massive amounts of high-value scenario training data to iteratively optimize models has become crucial.

[0004] Currently, autonomous driving manufacturers build dedicated closed-loop data platforms to collect training data and deploy training databases. During the training data collection process, different data mining requirements often arise. This requires the closed-loop data platform to collect large amounts of training data and clearly define the categories of the training data. Therefore, various classification models (classifiers) need to be trained and deployed on the platform to determine the categories of the collected training data and incorporate them into the training database.

[0005] In the above process, training and deploying different classification models on the data closed-loop platform not only requires a large amount of time and manpower costs, but also limits the categories of training data that can be collected by the data closed-loop platform.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a data collection method and related equipment, which greatly reduces manual participation in the data collection process, effectively saves time and labor costs, and can collect various categories of training data, which is conducive to improving the training database in the system.

[0008] A first aspect of an embodiment of the present application provides a data collection method, the method comprising:

[0009] The first terminal device receives a first feature from the network device, the first feature indicating a first scene; the first terminal device obtains a second image and processes the second image to obtain a second feature indicating a second scene. It should be noted that the second image is an image of a certain category, that is, the second scene is a scene of a certain category. Similarly, the first scene is also a scene of a certain category. The categories of the first and second scenes may be the same or different.

[0010] The first terminal device determines whether the first image and the second image belong to the same category of images based on the first feature and the second feature, that is, determines whether the first scene and the second scene belong to the same category of scenes. If the first scene and the second scene belong to the same category of scenes, the first terminal device sends the second image to the network device.

[0011] Based on the above technical solution, when collecting data, the terminal device receives the first feature from the network device, that is, the terminal device can collect data that matches the first feature based on the first feature, thereby realizing targeted data mining. At the same time, the collection process greatly reduces manual participation, which can effectively save time and labor costs.

[0012] In one possible implementation, the first terminal device may calculate the similarity between the first feature and the second feature, and determine whether the similarity between the first feature and the second feature is greater than or equal to a similarity threshold. If the similarity between the first feature and the second feature is greater than or equal to the similarity threshold, the first terminal device may determine that the first image and the second image are images of the same category, that is, the first scene and the second scene are scenes of the same category. Therefore, the first terminal device may determine that the second image is usable data (target data) and send the second image to the network device. If the similarity between the first feature and the second feature is less than the similarity threshold, that is, the first scene and the second scene are scenes of different categories, the terminal device may determine that the second image is unusable data (non-target data).

[0013] In one possible implementation, the terminal device first obtains a second image and encodes the second image through a third model to obtain a second feature, where the second feature indicates a second scene. The terminal device then detects whether the similarity between the first feature and the second feature is greater than or equal to a similarity threshold. If so, the terminal device sends the second image to the network device. In this implementation, the terminal device can deploy a multimodal basic model library, wherein the multimodal basic model library includes a variety of basic neural network processing models, such as the third model mentioned above. By using a universal and basic model library, it is possible to avoid developing and deploying different algorithms and models for different scenarios or data, thereby reducing human development and deployment costs.

[0014] In a possible implementation, the first feature is a text feature.

[0015] In one possible implementation, the first terminal device receives a first characteristic from a network device, further comprising: the first terminal device receiving a first parameter from the network device, the first parameter being used to indicate a frame rate at which the first terminal device will subsequently transmit images to the network device. In the aforementioned implementation, the network device may also transmit the first parameter to the terminal device simultaneously with the first characteristic, the first parameter being used to indicate a frame rate at which the terminal device will subsequently transmit images to the network device. Then, after determining that the second image is usable data, the terminal device may transmit the second image to the network device at the frame rate indicated by the first parameter.

[0016] In one possible implementation, the method further includes: the first terminal device receiving a second parameter from the network device, the second parameter being used to adjust the frame rate at which the first terminal device transmits images to the network device; and the first terminal device transmitting a third image to the network device based on the second parameter. In the aforementioned implementation, although the second image is determined by the terminal device to be usable data, the network device may further verify the second image to ultimately determine whether the second image is usable data. After the network device obtains the verification result, the network device may generate a second parameter and transmit the second parameter to the terminal device, the second parameter being used to adjust the frame rate at which the terminal device subsequently transmits images to the network device. If the verification result indicates that the second image is usable data, the second parameter is used to increase the frame rate at which the terminal device subsequently transmits images to the network device. Therefore, if a third image subsequently obtained by the terminal device is of the same category as the first image, the terminal device may increase the frame rate indicated by the first parameter based on the second parameter and transmit the third image to the network device at the increased frame rate. If the verification result shows that the second image is unusable data, the second parameter is used to reduce the frame rate of the subsequent images sent by the terminal device to the network device. Therefore, when the third image subsequently obtained by the terminal device is not of the same category as the first image, the terminal device can reduce the frame rate indicated by the first parameter based on the second parameter and send the third image to the network device at the reduced frame rate. It can be seen that the network device can perform a secondary verification on the target data (that is, the second image) uploaded by the terminal device, and thus flexibly adjust the rate at which the terminal device uploads subsequent data based on the result of the secondary verification. This allows the terminal device to reserve sufficient bandwidth to provide other services to the user, thereby improving the user experience.

[0017] In one possible implementation, the method further includes: the first terminal device receiving a third parameter from the network device, the third parameter indicating the network device's verification result of the second image, and sending the third image to the network device based on the verification result. In the aforementioned implementation, after the network device receives the verification result, the network device may generate a third parameter indicating the verification result and send the third parameter to the terminal device. The third parameter indicates the verification result. The terminal device may adjust the frame rate of subsequent images sent to the network device based on the verification result indicated by the third parameter. For example, if the verification result indicates that the second image is usable data, the terminal device may automatically increase the frame rate indicated by the first parameter and send the third image to the network device at the increased frame rate. If the verification result indicates that the second image is unusable data, the terminal device may automatically decrease the frame rate indicated by the first parameter and send the third image to the network device at the decreased frame rate. Thus, the terminal device can flexibly adjust the frame rate of subsequent data uploads based on the network device's secondary verification result. This allows the terminal device to flexibly control upload bandwidth and reserve sufficient bandwidth for other user services, thereby improving the user experience.

[0018] In one possible implementation, after this method, the method further includes: the first terminal device receiving a fourth parameter from the network device and adjusting the similarity threshold based on the fourth parameter. In the aforementioned implementation, after the network device obtains the verification result, the network device can generate a fourth parameter and send the fourth parameter to the terminal device, and the fourth parameter is used to adjust the aforementioned similarity threshold. If the verification result indicates that the second image is usable data, the fourth parameter is used to indicate that the terminal device can appropriately reduce the similarity threshold. Therefore, the terminal device reduces the current similarity threshold and collects the third image based on the reduced similarity threshold, so that more data can be collected; if the verification result indicates that the second image is unusable data, the fourth parameter is used to appropriately increase the similarity threshold. Therefore, the terminal device increases the current similarity threshold and collects the third image based on the increased similarity threshold. This can improve the accuracy of the data sent by the terminal device to the network device. It can be seen that the terminal device flexibly controls the threshold for collecting subsequent data based on the verification result, which enables the terminal device to improve data collection efficiency.

[0019] In one possible implementation, prior to the method, the method further includes: the second terminal device sending a first image to the network device based on a preset condition, where the first image is an image of the first scene. In the aforementioned implementation, the first feature sent by the network device is derived from the first image sent by the second terminal device. It should be understood that the second terminal device and the first terminal device can be the same or different terminal devices.

[0020] In one possible implementation, the preset condition includes: a second terminal device performs a first operation for a first scenario, and the second terminal device or user performs a second operation for the first scenario, where the first and second operations are different. For example, the preset condition may be: in autonomous driving, the first operation is performed by the autonomous driving system, and the second operation is performed by the user during human driving, where the autonomous driving system and the user perform different operations for the same scenario. For example, the preset condition may be: the first and second operations are both performed by the autonomous driving system for the same scenario, but the first and second operations are different. The preset condition may be described as when the terminal device discovers difficult data. Thus, the terminal device can independently mine various categories of difficult data (i.e., the first image), which is equivalent to the terminal device independently formulating data mining requirements and notifying the network device of the data mining requirements. In this way, the data acquisition system composed of the network device and the terminal device can automatically complete the collection and mining of corresponding training data according to the data mining requirements. Since the requirement formulation process does not require human intervention, it not only reduces costs but also improves the efficiency and accuracy of data acquisition.

[0021] A second aspect of an embodiment of the present application provides a data collection method, which includes: a network device sends a first feature to a first terminal device, where the first feature indicates a first scene; the network device receives a second image sent by the first terminal device, and processes the second image to obtain a third feature, where the third feature indicates a second scene; the network device generates a verification result of whether the second scene and the first scene belong to the same category of scenes based on the first feature and the third feature; the network device sends a second parameter and / or a third parameter to the first terminal device, where the second parameter is used to adjust the frame rate at which the first terminal device sends images to the network device, and the third parameter is used to indicate the verification result.

[0022] In a possible implementation, the network device sends the first feature to the first terminal device, further comprising: the network device sends a first parameter to the first terminal device, where the first parameter is used to indicate a frame rate at which the first terminal device subsequently sends images to the network device.

[0023] In a possible implementation manner, after the method, the method further includes: the network device sends a fourth parameter to the first terminal device, where the fourth parameter is used to adjust the similarity threshold of the first terminal device.

[0024] In a possible implementation manner, before the method, the method further includes: the network device receives a first image sent by the second terminal device, and obtains a first feature based on the first image.

[0025] In one possible implementation, a network device is deployed with a multimodal base model library, wherein the multimodal base model library includes a first model and a second model. For example, the network device may input a first image into the first model to convert the first image using the first model, thereby obtaining first text describing a first scene. The network device may then input the first text into the second model to encode the first text using the second model, thereby obtaining a first feature.

[0026] In a possible implementation, obtaining the first feature based on the first image includes: expanding the first image to obtain an image set, where the image set includes one or more images; and processing the image set to obtain the first feature.

[0027] A third aspect of an embodiment of the present application provides a terminal device, including: a receiving module for receiving a first feature from a network device, the first feature indicating a first scene; a processing module for processing a second image to obtain a second feature, the second feature indicating a second scene; and a sending module for determining, based on the first feature and the second feature, if the first scene and the second scene belong to the same category of scenes, and then sending the second image to the network device.

[0028] In a possible implementation, the sending module is configured to send the second image to the network device if it is determined that the similarity between the first feature and the second feature is greater than or equal to a similarity threshold.

[0029] In a possible implementation, the receiving module is further configured to receive a first parameter from the network device, where the first parameter is used to indicate a frame rate at which the terminal device subsequently sends images to the network device.

[0030] In a possible implementation, the sending module is configured to send the second image to the network device based on the first parameter.

[0031] In one possible implementation, the receiving module is further used to receive a second parameter from the network device, where the second parameter is used to adjust the frame rate at which the terminal device sends images to the network device; the sending module is further used to send a third image to the network device based on the second parameter.

[0032] In a possible implementation, the receiving module is further configured to receive a third parameter from the network device, where the third parameter is used to indicate a verification result of the second image by the network device; and the processing module is further configured to send the third image to the network device based on the verification result.

[0033] In a possible implementation, the receiving module is further configured to receive a fourth parameter from the network device; and the processing module is further configured to adjust the similarity threshold based on the fourth parameter.

[0034] In a possible implementation, the sending module is further configured to send the first image to the network device based on a preset condition.

[0035] In a possible implementation, the preset condition includes: the terminal device performs a first operation for a first scenario, the terminal device or the user performs a second operation for the first scenario, and the first operation and the second operation are different.

[0036] A fourth aspect of an embodiment of the present application provides a network device, comprising: a sending module for sending a first feature to a first terminal device, the first feature indicating a first scene; a receiving module for receiving a second image sent by the first terminal device, and processing the second image to obtain a third feature, the third feature indicating a second scene; a processing module for generating a verification result of whether the second scene and the first scene belong to the same category of scenes based on the first feature and the third feature; the sending module is also used to send a second parameter and / or a third parameter to the first terminal device, the second parameter being used to adjust the frame rate at which the first terminal device sends images to the network device, and the third parameter being used to indicate the verification result.

[0037] In a possible implementation, the sending module is further configured to send a first parameter to the first terminal device, where the first parameter is used to indicate a frame rate at which the first terminal device subsequently sends images to the network device.

[0038] In a possible implementation, the sending module is further configured to send a fourth parameter to the first terminal device, where the fourth parameter is used to adjust a similarity threshold of the first terminal device.

[0039] In a possible implementation, the receiving module is further configured to receive a first image sent by the second terminal device; and the processing module is further configured to obtain the first feature based on the first image.

[0040] In a possible implementation, the processing module is configured to expand the first image to obtain an image set, where the image set includes one or more images; and process the image set to obtain the first feature.

[0041] A fifth aspect of an embodiment of the present application provides a terminal device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the terminal device executes the method described in the first aspect or any possible implementation method of the first aspect.

[0042] A sixth aspect of an embodiment of the present application provides a network device, comprising a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the network device executes the method described in the second aspect or any possible implementation method of the second aspect.

[0043] A seventh aspect of an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0044] An eighth aspect of an embodiment of the present application provides a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes a method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0045] In a possible implementation, the processor is coupled to the memory through an interface.

[0046] In one possible implementation, the chip system further includes a memory, in which a computer program or computer instructions is stored.

[0047] A ninth aspect of an embodiment of the present application provides a computer storage medium storing a computer program, which, when executed by a computer, enables the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0048] A tenth aspect of an embodiment of the present application provides a computer program product, which stores instructions that, when executed by a computer, enable the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0049] An eleventh aspect of an embodiment of the present application provides a vehicle, the vehicle comprising a terminal device as described in the third aspect, any possible implementation of the third aspect, or the fifth aspect.

[0050] In an embodiment of the present application, when a network device and a terminal device jointly collect data, the terminal device may send a first image belonging to a first scene to the network device. The network device may then input the first image into a first model to convert the first image using the first model, thereby obtaining a first text describing the first scene. The network device may then input the first text into a second model to encode the first text using the second model, thereby obtaining a first feature, and send the first feature to the terminal device. Subsequently, the terminal device may encode the second image using a third model to obtain a second feature, which indicates a second scene, and detect whether the similarity between the first feature and the second feature is greater than or equal to a similarity threshold. If so, the terminal device sends the second image to the network device. Finally, the network device may classify the second image and the first image as training data for the same scene category for subsequent model training. In the aforementioned process, a multimodal basic model library is deployed in a data collection system comprising the network device and the terminal device, wherein the first and second models in the multimodal basic model library are deployed in the network device, and the third model in the multimodal basic model library is deployed in the terminal device. Then, the network device can extract the first feature of the first image based on the first model and the second model, and the terminal device can extract the second feature of the second image based on the third model. Therefore, the terminal device can determine whether the first image and the second image are images of the same category (that is, whether the first scene and the second scene are scenes of the same category) by judging the similarity between the first feature and the second feature. If so, the terminal device determines the second image as available training data and uploads it to the network device for storage. It can be seen that the data acquisition system can use the first image as a template for collecting training data through the multimodal basic model library, and automatically collect training data (second image, etc.) that belongs to the same category as the first image. The data collection process greatly reduces manual participation and can effectively save time and labor costs. In addition, since the category of the first image is not restricted, the data acquisition system can successfully collect training data of various categories, which is conducive to enriching and improving the training database in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;

[0052] FIG2a is a schematic structural diagram of a data acquisition system provided in an embodiment of the present application;

[0053] FIG2 b is a schematic diagram of a data acquisition device according to an embodiment of the present application;

[0054] FIG3 is a schematic diagram of the architecture of the system 100 provided in an embodiment of the present application;

[0055] FIG4 is a flow chart of a data acquisition method according to an embodiment of the present application;

[0056] FIG5a is a flow chart of a data acquisition system provided in an embodiment of the present application;

[0057] FIG5 b is another schematic diagram of a flow chart of a data acquisition system provided in an embodiment of the present application;

[0058] FIG5c is another schematic diagram of a flow chart of a data acquisition system provided in an embodiment of the present application;

[0059] FIG6 is another flow chart of the data acquisition method provided in an embodiment of the present application;

[0060] FIG7 is another flow chart of the data acquisition method provided in an embodiment of the present application;

[0061] FIG8 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

[0062] FIG9 is a schematic diagram of a structure of a network device provided in an embodiment of the present application;

[0063] FIG10 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

[0064] FIG11 is a schematic structural diagram of a network device provided in an embodiment of the present application;

[0065] FIG12 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The embodiments of the present application provide a data collection method and related equipment, which greatly reduces manual participation in the data collection process, effectively saves time and labor costs, and can collect various categories of training data, which is conducive to improving the training database in the system.

[0067] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0068] In the autonomous driving field, to continuously optimize autonomous driving models, autonomous driving manufacturers need to deploy dedicated training databases. Acquiring and mining difficult data is a key step in deploying these training databases. With the accelerated rollout of autonomous driving technology, the ability to automatically and efficiently collect massive amounts of training data to iteratively optimize models has become crucial.

[0069] Currently, autonomous driving manufacturers build dedicated closed-loop data platforms to collect training data and deploy training databases. During data collection, different data mining requirements often arise. This requires the closed-loop data platform to collect large amounts of training data and clearly define the categories of the training data. Consequently, various classification models (classifiers) need to be trained and deployed on the platform to determine the categories of the collected training data and incorporate them into the training database.

[0070] In the above process, training and deploying different classification models on the data closed-loop platform not only requires a large amount of time and manpower costs, but also limits the categories of training data that can be collected by the data closed-loop platform.

[0071] Furthermore, data closed-loop platforms are often deployed on network devices and are unable to determine various data mining requirements on their own. Data mining requirements need to be specified manually, which requires a lot of manual participation. Once there is too much manual participation, not only will costs increase, but it will also affect the efficiency and accuracy of data collection.

[0072] Furthermore, the terminal device uploads the collected data to the data closed-loop platform, generally at a fixed frame rate, which lacks flexibility. It also does not consider whether the uploaded data meets the requirements (or the availability of the uploaded data), which may waste the bandwidth of the terminal device.

[0073] In order to solve the above problems, the embodiments of the present application provide a data collection method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by sensing the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for data processing is a common application of artificial intelligence.

[0074] First, we will describe the overall workflow of an AI system. See Figure 1, which illustrates a schematic diagram of the AI ​​framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0075] (1) Infrastructure

[0076] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0077] (2) Data

[0078] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0079] (3) Data processing

[0080] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0081] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0082] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0083] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0084] (4) General ability

[0085] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0086] (5) Smart products and industry applications

[0087] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0088] Next, several application scenarios of this application are introduced.

[0089] Figure 2a is a structural diagram of a data acquisition system provided in an embodiment of the present application, which includes a terminal device and a data processing device (network device). Among them, the terminal device includes a mobile phone, a personal computer, a smart car (or an on-board terminal, a domain controller, a computing platform, etc.), or an intelligent terminal such as a data acquisition center. The data processing device can serve as the initiator of data acquisition and as the initiator of the data acquisition request. The data processing device usually initiates a request to the terminal device so that the terminal device completes the acquisition of training data based on the data acquisition request.

[0090] The aforementioned data processing devices can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing devices receive data collection requests from smart terminals via interactive interfaces. They then use data storage and data processing processors to perform data collection through machine learning, deep learning, search, reasoning, and decision-making. The memory in a data processing device is a general term that includes local storage and databases that store historical data. The database can be located on the data processing device or on other network servers.

[0091] In the data collection system shown in Figure 2a, the terminal device can receive instructions from the data processing device. For example, the terminal device can obtain a data collection request from the data processing device, and then, based on the data collection request, work with the data processing device to complete data collection and upload the collected data to the data processing device, so that the data processing device can build a database. For example, after receiving the data collection request from the data processing device, the terminal device can collect difficult data and upload the difficult data to the data processing device. The data processing device can then process the difficult data and send the features of the difficult data to the terminal device. The terminal device can then collect target data and extract features of the target data. By comparing the features of the difficult data with the features of the target data, it can determine whether the target data is available. If it is determined that the target data is available, the target data is uploaded to the data processing device, so that the data processing device can classify the target data as data belonging to the same category or scenario as the difficult data.

[0092] In FIG2 a , the data processing device and the terminal device can be combined to jointly execute the data collection method of the embodiment of the present application.

[0093] FIG2 b is a schematic diagram of data acquisition related equipment provided in an embodiment of the present application.

[0094] The terminal device in Figure 2a above can specifically be the local device 301 or the local device 302 in Figure 2b, and the data processing device in Figure 2a can specifically be the execution device 210 in Figure 2b, wherein the data storage system 250 can store the data to be processed of the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or it can be set on the cloud or other network servers.

[0095] The processor of the execution device or local device in Figure 2b can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned by the data to complete the data collection application, thereby collecting corresponding data.

[0096] Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of the present application. In Figure 3, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The execution device 110 can input data to the client device 140 through the I / O interface 112. The input data may include: data acquisition requests and other control parameters in the embodiment of the present application.

[0097] Then, the terminal device 140 can discover and collect difficult example data based on the input data and upload it to the execution device 110. When the execution device 110 pre-processes the difficult example data, or when the calculation module 111 of the execution device 110 performs calculations and other related processing (such as implementing the functions of the neural network model in this application), the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the difficult example data and the characteristics of the difficult example data obtained by the corresponding processing in the data storage system 150.

[0098] In addition, the execution device 110 may also send the features of the difficult example data to the terminal device 140 via the I / O interface 112. The terminal device 140 may further collect target data based on the features sent by the execution device 110. For example, the terminal device 140 obtains the target data and extracts the features of the target data. By comparing the features of the difficult example data with the features of the target data, it may determine whether the target data is available based on the comparison results. If it is determined that the target data is available, the terminal device 140 may send the target data to the execution device 110. In this way, the execution device 110 may store the obtained target data, etc. in the data storage system 150. It should be noted that the difficult example data and the target data stored in the data storage system 150 can be used as training data of the same category, retrieved by the execution device, and complete the training of other neural network models.

[0099] It is worth noting that Figure 3 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0100] The following is a brief description of the data collection method provided by this application.

[0101] The data collection method provided in the embodiment of the present application can be implemented through a data collection system, which includes a network device and a terminal device, and the two can jointly execute the data collection method provided in the embodiment of the present application, and the collected data can be used for model training on the network device side. Among them, both the network device and the terminal device can be deployed with a certain number of neural network models (pre-trained neural network models). Through these neural network models, the network device and the terminal device can process difficult case data and find target data associated with difficult case data to include the target data as available training data, thereby improving the training database. Then, the subsequent network device can use the training data in the training database to train other neural network models to be trained or continue to fine-tune the pre-trained neural network models on the network device and the terminal device.

[0102] Furthermore, the data collection method provided in the embodiments of this application can be applied in a variety of fields. One preferred field is autonomous driving. Accordingly, the network device can be a cloud server, and the terminal device can be a vehicle, an onboard terminal, a domain controller, a computing platform, etc. Together, they can form a vehicle-cloud collaborative data collection system to mine and collect training data.

[0103] To further understand the workflow of the data acquisition system provided in the embodiment of the present application, the following further describes the workflow in conjunction with FIG4 . FIG4 is a flow chart of a data acquisition method provided in the embodiment of the present application. As shown in FIG4 , the method includes:

[0104] 401. A terminal device sends a first image belonging to a first scene to a network device.

[0105] In this embodiment, when the network device and the terminal device need to jointly collect data, the terminal device may first obtain a first image for presenting the first scene (the content presented by the first image is the first scene, which can also be understood as the first image belonging to the first scene), and send the first image to the network device. It should be noted that the first image (also referred to as difficult data) is usually an image of a certain category discovered and collected by the terminal device, that is, the first scene is a scene of a certain category discovered and decided to be collected by the terminal device. For example, the first scene can be a scene of small animals on the road in the field of autonomous driving. This scene is one of the scenes in the field of autonomous driving. The first scene can also be other scenes in the field of autonomous driving, such as a scene of a vehicle passing through a tunnel, etc. The first scene can also be a scene in other fields. It can be seen that different scenes in the same field can be regarded as scenes of different categories, and scenes in different fields can also be regarded as scenes of different categories.

[0106] Specifically, the terminal device may obtain and send the first image to the network device in the following manner:

[0107] (1) When the terminal device is in the first scene, the terminal device may input the first image used to present the first scene into the ninth model (a certain operation prediction model, which is a trained neural network model) to process the first image through the ninth model, thereby obtaining a first operation for the first scene. Still as in the above example, assuming that the first scene is a scene of small animals on the road in the field of autonomous driving, when the user encounters a scene of a black dog on the road while driving the vehicle, the vehicle-mounted terminal may also input the image used to present the scene of a black dog on the road into a certain operation prediction model to obtain a driving operation 1 for the scene of a black dog on the road.

[0108] (2) When the terminal device is in the first scene, the terminal device can also obtain the second operation for the first scene. The second operation can be obtained in a variety of ways: (2.1) When the terminal device is in the first scene, it can collect information about the user in real time (through sensors), thereby obtaining the user's second operation for the first scene. For example, when a user is driving a vehicle and encounters a scene with a black dog on the road, the user can make a driving operation for the scene, and the vehicle-mounted terminal can capture the user's driving operation 2 for the scene with a black dog on the road in real time. (2.2) When the terminal device is in the first scene, the terminal device can input the first image used to present the first scene into the tenth model (another operation prediction model, which is a trained neural network model) to process the first image through the tenth model, thereby obtaining the second operation for the first scene. Still like the above example, the vehicle-mounted terminal can input the image used to present the scene with a black dog on the road into another operation prediction model to obtain the driving operation 2 for the scene with a black dog on the road.

[0109] (3) After obtaining the first operation for the first scene and the second operation for the first scene, the terminal device can detect whether the first operation for the first scene and the second operation for the first scene are the same. If the first operation for the first scene and the second operation for the first scene are different, the terminal device can determine the first image used to present the first scene as difficult example data and send the first image to the network device. As in the above example, if the vehicle terminal determines that driving operation 1 and driving operation 2 are different operations, the image used to present the scene of a black dog on the road is determined to be a difficult example image and uploaded to the cloud server.

[0110] More specifically, the terminal device may also obtain and send the first image to the network device in the following manner:

[0111] The terminal device cannot obtain the operation for the first scene, that is, the model on the terminal device cannot derive the operation for the first scene for the first image. For example, the model has not been trained for the first scene or the training effect is poor, which may require user takeover, etc. At this time, the terminal device can determine the first image used to present the first scene as difficult example data, and send the first image to the network device, etc.

[0112] It should be understood that in the present application, the terminal device collects data or images through its own collection capabilities. For example, the terminal device is a vehicle, and the vehicle includes various on-board sensors, such as image sensors (such as cameras), radars (millimeter wave radars, lidars, ultrasonic radars), etc., and images and data can be collected through on-board sensors (such as image sensors, etc.); it can also be that the terminal device controls other devices to collect data, for example, the terminal device is a vehicle-mounted domain controller or computing platform, and the vehicle-mounted domain controller can control the vehicle-mounted sensors to collect data or images, etc.

[0113] 402. The network device processes the first image to obtain a first feature, where the first feature is used to indicate a first scene.

[0114] After obtaining the first image, the network device can obtain the first model (image-text conversion model, which is a trained neural network model) and the second model (text encoder, which is a trained neural network model) from the multimodal basic model library. Then, the network device can input the first image into the first model to convert the first image through the first model, thereby obtaining a first text for describing the first scene. Then, the network device can input the first text into the second model to encode the first text through the second model, thereby obtaining a first feature (which can be a text feature). Since the first image is used to present the first scene, the first feature obtained based on the first image can be used to indicate the first scene.

[0115] Continuing with the above example, as shown in FIG5a (FIG5a is a flow chart of a data acquisition system provided by an embodiment of the present application), after obtaining an image representing a scene of a black dog on the road, the cloud server can select an image-to-text conversion model (e.g., egBLIP2, miniGPT4, etc.) from the multimodal basic model library to process the image to implement image2text, thereby obtaining text describing the scene of a puppy on the road, i.e., "a photo of a dog on the road." The cloud server can then select a text encoder from the multimodal basic model library to process the text to extract features, thereby obtaining text features.

[0116] Specifically, the network device may obtain the first feature in the following manner:

[0117] (1) After obtaining the first image, the network device may obtain a first model (an image-to-text conversion model, which is a trained neural network model) and a fifth model (an image-to-image enhancement model, which is a trained neural network model) from the multimodal basic model library. The network device may then input the first image into the first model to convert the first image using the first model, thereby obtaining a first text describing the first scene.

[0118] (2) After obtaining the first image and the first text, the network device may further input the first image into the fifth model to enhance (expand) the first image through the fifth model to obtain an image set, wherein the image set includes one or more images. The first scene is associated with the scene to which the images in the image set belong. That is, the first scene and the scene to which the images in the image set belong are scenes of the same category. For example, when the first scene is a scene of small animals on the road in the field of autonomous driving, the scene to which the images in the image set belong is also a scene of small animals on the road in the field of autonomous driving.

[0119] Then, after obtaining the first text and image set, the network device can use the first text and image set to train the second model and the fourth model (mapping model, trained neural network model), and add the second model and the fourth model to the multimodal basic model library.

[0120] (3) Then, the network device may input the first text into the fourth model to map the first text through the fourth model, thereby obtaining an intermediate feature. Subsequently, the network device may input the intermediate feature into the second model to encode the intermediate feature through the second model, thereby obtaining a first feature (which may also be understood as a text feature).

[0121] Still taking the above example, as shown in Figures 5b and 5c (Figure 5b is another flow chart of the data acquisition system provided in an embodiment of the present application, and Figure 5c is another flow chart of the data acquisition system provided in an embodiment of the present application, and Figures 5b and 5c are drawn based on Figure 5a), after obtaining an image for presenting a scene with a black dog on the road, the cloud server can select an image-to-text conversion model (such as egBLIP2, miniGPT4, etc.) from the multimodal basic model library to process the image to implement image2text, thereby obtaining text for describing the scene with a puppy on the road, namely "a photo of a dog on the road".

[0122] Then, the cloud server can also select an image-image enhancement model (for example, egTag2text+Detclip+SAM+SD) from the multimodal basic model library to process the image to achieve image2image, thereby obtaining a new image for presenting a scene with a bird on the road, a new image for presenting a scene with a kitten on the road, and a new image for presenting a scene with a deer on the road, and so on.

[0123] The cloud server then selects two pre-trained base models from the multimodal base model library and trains (fine-tunes) them using the aforementioned text and new images to obtain a mapping model and a text encoder. The cloud server then processes the aforementioned text using the mapping model to obtain intermediate features, and then processes the intermediate features using the text encoder to obtain text features.

[0124] It's worth noting that the original multimodal base model library includes multiple pre-trained base models. To adapt the entire model library to the autonomous driving domain, these base models can be fine-tuned using autonomous driving data, resulting in multiple fine-tuned base models. These fine-tuned base models comprise the aforementioned multimodal base model library. These fine-tuned base models include the aforementioned image-to-text conversion model, the aforementioned text encoder, the aforementioned image-to-image enhancement model, the aforementioned mapping model, the subsequent visual encoder, the subsequent large-scale image-to-text conversion model, the subsequent large-scale text encoder, the subsequent large-scale visual encoder, and so on. Because these models are fine-tuned using autonomous driving data, the outputs of these models (e.g., the aforementioned text, intermediate features, text features, and new images) contain more details and features relevant to various scenarios in the autonomous driving domain. This makes the entire data acquisition system more adaptable to the autonomous driving domain. Of course, if the data acquisition system is applied to other domains, model fine-tuning can also be performed using data from other domains, or the pre-trained base models (i.e., the general base models) can be used directly to complete the training data collection.

[0125] 403. The network device sends the first feature to the terminal device.

[0126] After obtaining the first feature, the network device may send the first feature to the terminal device.

[0127] 404. The terminal device processes the second image belonging to the second scene to obtain a second feature, where the second feature is used to indicate the second scene.

[0128] After obtaining the first feature, the terminal device can capture a second image used to present the second scene, obtain a third model from the multimodal basic model library, and then input the second image into the third model (a visual encoder, which is a trained neural network model) to encode the second image through the third model, thereby obtaining a second feature (which can also be understood as a visual feature). Since the second image is used to present the second scene, the second feature obtained based on the second image can be used to indicate the second scene.

[0129] As in the previous example, after obtaining text features, the cloud server can send them to the vehicle terminal. After obtaining the text features, the vehicle terminal can capture an image representing a scene (e.g., a chicken on the road) in real time while the vehicle is driving and identify the image as the target image to be detected. The vehicle terminal can then process the image using a visual encoder to obtain visual features.

[0130] 405. The terminal device determines, based on the first feature and the second feature, that if the first scene and the second scene belong to the same category of scenes, then the terminal device sends the second image to the network device.

[0131] After obtaining the second feature, the terminal device can calculate the first feature and the second feature to obtain the similarity between the first feature and the second feature. Then, the terminal device can determine whether the similarity between the first feature and the second feature is greater than or equal to a similarity threshold (the similarity threshold represents the benchmark value for the image obtained by the terminal device to achieve usability or target. It can be set according to actual needs. It can be preset by the terminal device or sent to the terminal device by the network device. The threshold can be unified, that is, applicable to all types of scenes, or it can correspond to the first feature or the scene indicated by the first feature, that is, different types of scenes can be different, which is not limited here). If the similarity between the first feature and the second feature is greater than or equal to the similarity threshold, the terminal device can determine that the first image and the second image are images of the same category, that is, the first scene and the second scene are scenes of the same category, so the terminal device can determine the second image as usable data and send it to the network device. If the similarity between the first feature and the second feature is less than the similarity threshold, the terminal device can determine that the first image and the second image are images of different categories, that is, the first scene and the second scene are scenes of different categories, so the terminal device can determine the second image as unusable data and discard the second image.

[0132] Specifically, the terminal device may send the second image to the network device in the following ways:

[0133] When the network device sends the first feature to the terminal device, it can also send the first parameter to the terminal device. The first parameter is used to indicate the frame rate (also called the upload frame rate) at which the terminal device subsequently sends images to the network device. When the terminal device determines that the second image is available data, it can send the second image to the network device based on the first parameter.

[0134] It can be understood that since the first parameter is used to indicate the frame rate at which the terminal device subsequently sends images to the network device, that is, the frame rate set by the network device for the terminal device, after the terminal device determines that the second image is available data, it can send the second image to the network device at the frame rate indicated by the first parameter.

[0135] As in the previous example, the cloud server can send text features and corresponding control parameters to the vehicle terminal, where the control parameters include the upload frame rate. Therefore, after the vehicle terminal determines that the similarity between the text features and the visual features is greater than or equal to the similarity threshold, it can upload the target image to the cloud server at the upload frame rate included in the control parameters.

[0136] 406. The network device determines the first image and the second image as training data for model training.

[0137] After obtaining the second image, the network device can classify the second image as data of the same scene category as the first image and store it in the training database. As the terminal device continues to upload subsequent images, the network device can also determine that the subsequent images belong to the same category of training data as the first image. This can continuously improve this type of training data in the training database and can be used for subsequent model training to train the corresponding neural network model.

[0138] Specifically, the network device may also perform secondary mining (secondary verification) on the second image to ensure the quality of the training data:

[0139] (1) Although the second image is determined by the terminal device as usable data, the network device may further verify the second image to finally determine whether the second image is usable data. Therefore, the network device may obtain the sixth model (large-scale image-text conversion model, which is a trained neural network model), the seventh model (large-scale text encoder, which is a trained neural network model) and the eighth model (large-scale visual encoder, which is a trained neural network model) from the multimodal basic model library. Then, the network device may input the first image into the sixth model to convert the first image through the sixth model, thereby obtaining a second text for describing the first scene. Then, the network device may input the second text into the seventh model to encode the second text through the seventh model, thereby obtaining a fourth feature (which can also be understood as a new text feature). Since the first image is used to present the first scene, the fourth feature obtained based on the first image can be used to indicate the first scene. Subsequently, the network device may also input the second image into the eighth model to encode the second image through the eighth model, thereby obtaining a third feature (which can also be understood as a new visual feature). Since the second image is used to present the second scene, the third feature obtained based on the second image can be used to indicate the second scene.

[0140] It should be noted that compared to the first model, the sixth model is larger and has better performance. Similarly, compared to the second model, the seventh model is larger and has better performance. Similarly, compared to the third model, the eighth model is larger and has better performance.

[0141] After obtaining the fourth feature and the third feature, the network device may calculate the similarity between the fourth feature and the third feature, and detect whether the similarity between the fourth feature and the third feature is greater than or equal to a similarity threshold.

[0142] If the similarity between the fourth feature and the third feature is greater than or equal to the similarity threshold, the network device may determine that the second image is available data. In this case, the network device determines the first image and the second image as training data belonging to the same category (that is, the network device determines the first scene and the second scene as scenes of the same category) and stores them in the same area of ​​the training database. If the similarity between the fourth feature and the third feature is less than the similarity threshold, that is, the second image does not meet the preset conditions, the network device may determine that the second image is unavailable data. In this case, the network device determines the first image and the second image as training data belonging to different categories (that is, the network device determines the first scene and the second scene as scenes of different categories) and stores them in different areas of the training database.

[0143] (2) Although the second image is determined by the terminal device as usable data, the network device may further verify the second image to ultimately determine whether the second image is usable data. Therefore, the network device may obtain the eighth model (large-size visual encoder, which is a trained neural network model) from the multimodal basic model library. Then, the network device may also input the second image into the eighth model to encode the second image through the eighth model, thereby obtaining a third feature (which can also be understood as a new visual feature). Since the second image is used to present the second scene, the third feature obtained based on the second image can be used to indicate the second scene.

[0144] If the similarity between the first feature and the third feature is greater than or equal to the similarity threshold, the network device may determine that the second image is available data. In this case, the network device determines the first image and the second image as training data belonging to the same category (that is, the network device determines the first scene and the second scene as scenes of the same category) and stores them in the same area of ​​the training database. If the similarity between the first feature and the third feature is less than the similarity threshold, the network device may determine that the second image is unavailable data. In this case, the network device determines the first image and the second image as training data belonging to different categories (that is, the network device determines the first scene and the second scene as scenes of different categories) and stores them in different areas of the training database.

[0145] It can be seen that the network device can obtain the verification result of the second image, which is used to indicate whether the first image and the second image are images of the same category or different categories, that is, to indicate whether the first scene and the second scene are scenes of the same category or different categories.

[0146] More specifically, the network device may further adjust the frame rate at which the terminal device sends subsequent images based on the availability of the second image:

[0147] (1) After the network device obtains the verification result, the network device may generate a second parameter and send the second parameter to the terminal device. The second parameter is used to adjust the frame rate of the image subsequently sent by the terminal device to the network device. If the verification result is used to indicate that the first scene and the second scene are scenes of the same category, the second parameter is used to increase the frame rate of the image subsequently sent by the terminal device to the network device. Therefore, when the third image subsequently acquired by the terminal device is an image of the same category as the first image (the third image belongs to the third scene, that is, the terminal device determines the first scene and the third scene to be scenes of the same category), the terminal device may increase the frame rate indicated by the first parameter based on the second parameter and send the third image to the network device at the increased frame rate. If the verification result is used to indicate that the first scene and the second scene are scenes of different categories, the second parameter is used to reduce the frame rate of the image subsequently sent by the terminal device to the network device. Therefore, when the third image subsequently acquired by the terminal device is an image of the same category as the first image, the terminal device may reduce the frame rate indicated by the first parameter based on the second parameter and send the third image to the network device at the reduced frame rate.

[0148] (2) After the network device obtains the verification result, the network device may generate a third parameter and send the third parameter to the terminal device, where the third parameter is used to indicate the verification result. If the terminal device determines that the first scene and the second scene are scenes of the same category based on the verification result indicated by the third parameter, once the third image subsequently acquired by the terminal device is an image of the same category as the first image, the terminal device may automatically increase the frame rate indicated by the first parameter and send the third image to the network device at the increased frame rate. If the terminal device determines that the first scene and the second scene are scenes of different categories based on the verification result indicated by the third parameter, once the third image subsequently acquired by the terminal device is an image of the same category as the first image, the terminal device may automatically reduce the frame rate indicated by the first parameter and send the third image to the network device at the reduced frame rate.

[0149] (3) In addition, after the network device obtains the verification result, the network device can generate a fourth parameter and send the fourth parameter to the terminal device. The fourth parameter is used to adjust the aforementioned similarity threshold. If the verification result is used to indicate that the first scene and the second scene are scenes of the same category, the fourth parameter is used to appropriately reduce the similarity threshold, so the terminal device reduces the current similarity threshold and collects the third image and subsequent images based on the reduced similarity threshold. By appropriately reducing the similarity threshold, the terminal device can collect more data and quickly supplement the training database. If the verification result is used to indicate that the first scene and the second scene are scenes of different categories, the fourth parameter is used to appropriately increase the similarity threshold, so the terminal device increases the current similarity threshold and collects the third image based on the increased similarity threshold. By increasing the similarity threshold, the accuracy of the available data obtained by the terminal device can be improved, avoiding waste of user bandwidth. It should be understood that the fourth parameter can also be obtained based on the secondary verification results of the second image sent by multiple terminal devices (such as the third terminal device, the fourth terminal device, etc.). By adjusting the similarity threshold based on the secondary verification results of the images sent by multiple terminal devices, the availability of the data obtained by the terminal device can be improved.

[0150] It should be understood that in the embodiment of the present application, the terminal device executing step 401 and the terminal device executing steps 404 and 405 may be the same terminal device, for example, the terminal devices executing steps 401, 404, and 405 are all first terminal devices. The terminal device executing step 401 and the terminal device executing steps 404 and 405 may be different terminal devices, for example, the terminal device executing step 401 is the second terminal device, and the terminal devices executing steps 404 and 405 are all first terminal devices. An example is as follows: the second terminal device sends a first image belonging to a first scene to the network device, the network device processes the first image to obtain a first feature, and sends the first feature to one or more terminal devices, which may include the first terminal device; the first terminal device obtains the second image, processes the second image to obtain a second feature, and determines whether the second image is target data based on the first feature and the second feature. If it is determined to be target data, the second image is sent to the network device; the network device can enter the second image into the database as scene data of the same type as the first image, or the network device can perform a secondary verification on the second image and send parameters based on the verification results (such as the subsequent upload frame rate of the terminal or the verification result, etc.). The above-mentioned first terminal device and second terminal device can be the same or different terminal devices.

[0151] FIG6 is another flow chart of a data acquisition method provided in an embodiment of the present application. As shown in FIG6 , the method includes:

[0152] 601. A first terminal device receives a first feature from a network device, where the first feature indicates a first scenario.

[0153] For an introduction to step 601 , please refer to the relevant description of steps 402 and 403 in the embodiment shown in FIG4 , which will not be repeated here.

[0154] 602. The first terminal device processes the second image to obtain a second feature, where the second feature indicates a second scene.

[0155] For an introduction to step 602 , please refer to the relevant description of step 404 in the embodiment shown in FIG. 4 , which will not be repeated here.

[0156] 603. The first terminal device determines, based on the first feature and the second feature, that if the first scene and the second scene belong to the same category of scenes, then the first terminal device sends the second image to the network device.

[0157] For an introduction to step 603 , please refer to the relevant description of step 405 in the embodiment shown in FIG. 4 , which will not be repeated here.

[0158] In one possible implementation, the first terminal device determines, based on the first feature and the second feature, that the second image and the first scene belong to the same category of scenes, and then the first terminal device sends the second image to the network device, including: the first terminal device determines that if the similarity between the first feature and the second feature is greater than or equal to a similarity threshold, then the first terminal device sends the second image to the network device. For an introduction to this implementation, please refer to the relevant description of step 405 in the embodiment shown in Figure 4, and will not be repeated here.

[0159] In one possible implementation, the first terminal device receiving the first characteristic from the network device further includes: the first terminal device receiving a first parameter from the network device, where the first parameter is used to indicate a frame rate at which the first terminal device subsequently transmits images to the network device. For an introduction to this implementation, please refer to the relevant description of step 405 in the embodiment shown in FIG. 4 , and will not be further elaborated here.

[0160] In one possible implementation, the first terminal device sending the second image to the network device includes: the first terminal device sending the second image to the network device based on the first parameter. For an introduction to this implementation, please refer to the relevant description of step 405 in the embodiment shown in Figure 4, and will not be repeated here.

[0161] In one possible implementation, after step 603, the process further includes: the first terminal device receiving a second parameter from the network device, where the second parameter is used to adjust the frame rate of images sent by the first terminal device to the network device; and the first terminal device sending a third image to the network device based on the second parameter. For an introduction to this implementation, please refer to the relevant description of step 406 in the embodiment shown in Figure 4 and will not be repeated here.

[0162] In one possible implementation, after step 603, the first terminal device further includes receiving a third parameter from the network device, the third parameter indicating the network device's verification result of the second image, and sending the third image to the network device based on the verification result. For an introduction to this implementation, please refer to the relevant description of step 406 in the embodiment shown in Figure 4 and will not be repeated here.

[0163] In one possible implementation, after step 603, the method further includes: the first terminal device receives a fourth parameter from the network device and adjusts the similarity threshold based on the fourth parameter. For an introduction to this implementation, please refer to the relevant description of step 406 in the embodiment shown in Figure 4, and will not be repeated here.

[0164] In one possible implementation, before step 601, the second terminal device sends the first image to the network device based on a preset condition. For an introduction to this implementation, please refer to the relevant description of step 401 in the embodiment shown in FIG4 , which will not be repeated here.

[0165] In one possible implementation, the preset condition includes: the second terminal device performs a first operation in response to the first scenario, and the second terminal device or user performs a second operation in response to the first scenario, where the first operation and the second operation are different. For an introduction to this implementation, please refer to the relevant description of step 401 in the embodiment shown in FIG. 4 , and will not be repeated here.

[0166] FIG7 is another flow chart of a data acquisition method provided in an embodiment of the present application. As shown in FIG7 , the method includes:

[0167] 701. The network device sends a first feature to a first terminal device, where the first feature indicates a first scenario.

[0168] For an introduction to step 701 , please refer to the relevant description of step 402 and step 403 in the embodiment shown in FIG4 , which will not be repeated here.

[0169] 702. The network device receives the second image sent by the first terminal device, and processes the second image to obtain a third feature, where the third feature indicates a second scene.

[0170] For an introduction to step 702 , please refer to the relevant description of step 406 in the embodiment shown in FIG. 4 , which will not be repeated here.

[0171] 703. The network device generates a verification result based on the first feature and the third feature to determine whether the second scene and the first scene belong to the same category of scenes.

[0172] For an introduction to step 703 , please refer to the relevant description of step 406 in the embodiment shown in FIG. 4 , which will not be repeated here.

[0173] 704. The network device sends a second parameter and / or a third parameter to the first terminal device, where the second parameter is used to adjust a frame rate at which the first terminal device sends images to the network device, and the third parameter is used to indicate a verification result.

[0174] For an introduction to step 704 , please refer to the relevant description of step 406 in the embodiment shown in FIG. 4 , which will not be repeated here.

[0175] In one possible implementation, the network device sending the first characteristic to the first terminal device further includes sending a first parameter to the first terminal device, where the first parameter is used to indicate a frame rate at which the first terminal device subsequently sends images to the network device. For an introduction to this implementation, please refer to the relevant description of step 405 in the embodiment shown in FIG. 4 , and will not be further elaborated here.

[0176] In one possible implementation, the method further includes: the network device sending a fourth parameter to the first terminal device, where the fourth parameter is used to adjust the similarity threshold of the first terminal device. For an introduction to this implementation, please refer to the relevant description of step 406 in the embodiment shown in Figure 4, and will not be repeated here.

[0177] In one possible implementation, before step 701, the method further includes: the network device receiving the first image sent by the second terminal device, and obtaining the first feature based on the first image. For an introduction to this implementation, please refer to the relevant description of step 402 in the embodiment shown in Figure 4, and will not be repeated here.

[0178] In one possible implementation, obtaining the first feature based on the first image includes: expanding the first image to obtain an image set, where the image set includes one or more images; and processing the image set to obtain the first feature. For an introduction to this implementation, please refer to the description of step 402 in the embodiment shown in FIG. 4 , and will not be repeated here.

[0179] In an embodiment of the present application, when a network device and a terminal device need to jointly collect training data, the terminal device may send a first image representing a first scene to the network device. The network device may then input the first image into a first model to transform the first image using the first model, thereby obtaining a first text describing the first scene. The network device may then input the first text into a second model to encode the first text using the second model, thereby obtaining a first feature, and send the first feature to the terminal device. Subsequently, the terminal device may encode a second image representing a second scene using a third model to obtain a second feature, and detect whether the similarity between the first feature and the second feature is greater than or equal to a similarity threshold. If so, the terminal device sends the second image to the network device. Finally, the network device may classify the second image and the first image as training data of the same category for subsequent model training. In the aforementioned process, a multimodal basic model library is deployed in a data acquisition system comprising the network device and the terminal device, wherein the first and second models in the multimodal basic model library are deployed in the network device, and the third model in the multimodal basic model library is deployed in the terminal device. Then, the network device can extract the first feature of the first image based on the first model and the second model, and the terminal device can extract the second feature of the second image based on the third model. Therefore, the terminal device can determine whether the first image and the second image are images of the same category (that is, whether the first scene and the second scene are scenes of the same category) by judging the similarity between the first feature and the second feature. If so, the terminal device determines the second image as available training data and uploads it to the network device for storage. It can be seen that the data acquisition system can use the first image as a template for collecting training data through the multimodal basic model library, and automatically collect training data (second image, etc.) that belongs to the same category as the first image. The data acquisition process greatly reduces manual participation and can effectively save time and labor costs. In addition, since the category of the first image is not restricted, the data acquisition system can successfully collect training data of various categories, which is conducive to enriching and improving the training database in the system.

[0180] Furthermore, in the data acquisition system provided by the embodiment of the present application, the terminal device can independently mine various categories of difficult example data (that is, the first image), which is equivalent to the terminal device independently formulating data mining requirements and notifying the network device of the data mining requirements. In this way, the data acquisition system composed of network devices and terminal devices can automatically complete the collection and mining of corresponding training data according to the data mining requirements (using difficult example data as a collection template for training data). Since the process of formulating requirements does not require human participation, it can not only reduce costs, but also improve the efficiency and accuracy of data collection.

[0181] Furthermore, in the data acquisition system provided in the embodiment of the present application, the network device can perform secondary mining on the target data (that is, the second image) uploaded by the terminal device, thereby flexibly controlling the rate at which the terminal device uploads subsequent data based on the results of the secondary mining. This allows the terminal device to reserve sufficient bandwidth to provide other services to the user, thereby improving the user experience.

[0182] The above is a detailed description of the data acquisition method provided by the embodiment of the present application. The following is an introduction to the data acquisition device provided by the embodiment of the present application, wherein the data acquisition device includes a terminal device and a network device. Figure 8 is a schematic diagram of the structure of the terminal device provided by the embodiment of the present application. As shown in Figure 8, the terminal device includes:

[0183] A receiving module 801 is configured to receive a first feature from a network device, where the first feature indicates a first scenario;

[0184] A processing module 802 is configured to process the second image to obtain a second feature, where the second feature indicates a second scene;

[0185] The sending module 803 is configured to send the second image to the network device if it is determined based on the first feature and the second feature that the first scene and the second scene belong to the same category of scenes.

[0186] In a possible implementation, the sending module 803 is configured to send the second image to the network device if it is determined that the similarity between the first feature and the second feature is greater than or equal to a similarity threshold.

[0187] In a possible implementation, the receiving module 801 is further configured to receive a first parameter from the network device, where the first parameter is used to indicate a frame rate at which the terminal device subsequently sends images to the network device.

[0188] In a possible implementation, the sending module 803 is configured to send the second image to the network device based on the first parameter.

[0189] In one possible implementation, the receiving module 801 is further used to receive a second parameter from the network device, where the second parameter is used to adjust the frame rate at which the terminal device sends images to the network device; the sending module 803 is further used to send a third image to the network device based on the second parameter.

[0190] In one possible implementation, the receiving module 801 is further configured to receive a third parameter from the network device, where the third parameter is used to indicate a verification result of the second image by the network device; and the processing module 802 is further configured to send the third image to the network device based on the verification result.

[0191] In a possible implementation, the receiving module 801 is further configured to receive a fourth parameter from a network device; and the processing module 802 is further configured to adjust the similarity threshold based on the fourth parameter.

[0192] In a possible implementation, the sending module 803 is further configured to send the first image to the network device based on a preset condition.

[0193] In a possible implementation, the preset condition includes: a terminal device performs a first operation for a first scenario, a second terminal device or a user performs a second operation for the first scenario, and the first operation and the second operation are different.

[0194] FIG9 is a schematic diagram of the structure of a network device provided in an embodiment of the present application. As shown in FIG9 , the network device includes:

[0195] A sending module 903 is configured to send a first feature to a first terminal device, where the first feature indicates a first scenario;

[0196] A receiving module 901 is configured to receive a second image sent by a first terminal device, and process the second image to obtain a third feature, where the third feature indicates a second scene;

[0197] A processing module 902 is configured to generate a verification result of whether the second scene and the first scene belong to the same category of scenes based on the first feature and the third feature;

[0198] The sending module 903 is further used to send a second parameter and / or a third parameter to the first terminal device, the second parameter is used to adjust the frame rate of the image sent by the first terminal device to the network device, and the third parameter is used to indicate the verification result.

[0199] In a possible implementation, the sending module 903 is further configured to send a first parameter to the first terminal device, where the first parameter is used to indicate a frame rate at which the first terminal device subsequently sends images to the network device.

[0200] In a possible implementation, the sending module 903 is further configured to send a fourth parameter to the first terminal device, where the fourth parameter is used to adjust a similarity threshold of the first terminal device, and the first terminal device sends the image to the network device based on the adjusted similarity threshold.

[0201] In a possible implementation, the receiving module 901 is further configured to receive a first image sent by the second terminal device; and the processing module 902 is further configured to obtain a first feature based on the first image.

[0202] In a possible implementation, the processing module 902 is configured to expand the first image to obtain an image set, where the image set includes one or more images; and process the image set to obtain a first feature.

[0203] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.

[0204] The embodiment of the present application also relates to a terminal device, and FIG10 is a structural diagram of the terminal device provided in the embodiment of the present application. As shown in FIG10 , the terminal device 1000 can be specifically manifested as a mobile phone, a tablet, a laptop computer, a smart wearable device, a car, a vehicle-mounted terminal, a vehicle-mounted domain controller, a vehicle-mounted computing platform, a server, etc., which are not limited here. Among them, the terminal device 1000 can be deployed with the device described in the embodiment corresponding to FIG8 to implement the function of data acquisition in the embodiment corresponding to FIG6 and / or the function of the terminal device in FIG4. Specifically, the terminal device 1000 includes: a receiver 1001, a transmitter 1002, a processor 1003 and a memory 1004 (wherein the number of processors 1003 in the terminal device 1000 can be one or more, and FIG10 takes one processor as an example), wherein the processor 1003 may include an application processor 10031 and a communication processor 10032. In some embodiments of the present application, the receiver 1001, the transmitter 1002, the processor 1003 and the memory 1004 can be connected via a bus or other means.

[0205] Memory 1004 may include read-only memory and random access memory, and provides instructions and data to processor 1003. A portion of memory 1004 may also include non-volatile random access memory (NVRAM). Memory 1004 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0206] Processor 1003 controls the operation of the terminal device. In specific applications, the various components of the terminal device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0207] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1003. Processor 1003 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1003. The above processor 1003 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1003 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1004, and processor 1003 reads the information in memory 1004 and, in conjunction with its hardware, completes the steps of the above method.

[0208] Receiver 1001 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the terminal device. Transmitter 1002 can be used to output digital or character information through the first interface. Transmitter 1002 can also be used to send instructions to the disk pack through the first interface to modify the data in the disk pack. Transmitter 1002 can also include a display device such as a display screen.

[0209] In an embodiment of the present application, in one case, the processor 1003 is used to cooperate with the network device to complete the collection of training data through the third model in the embodiment corresponding to Figure 4.

[0210] The embodiment of the present application also relates to a network device, and Figure 11 is a structural diagram of the network device provided in the embodiment of the present application. As shown in Figure 11, the network device 1100 can be specifically manifested as a mobile phone, a tablet, a laptop computer, an intelligent wearable device, a server, etc., which is not limited here. Among them, the network device 1100 can be deployed with the device described in the embodiment corresponding to Figure 9 to implement the function of data acquisition in the embodiment corresponding to Figure 7 and / or the function of the network device in Figure 4. Specifically, the network device 1100 includes: a receiver 1101, a transmitter 1102, a processor 1103 and a memory 1104 (wherein the number of processors 1103 in the network device 1100 can be one or more, and Figure 11 takes one processor as an example), wherein the processor 1103 may include an application processor 11031 and a communication processor 11032. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 1103 and the memory 1104 can be connected via a bus or other means.

[0211] The memory 1104 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1103. A portion of the memory 1104 may also include non-volatile random access memory (NVRAM). The memory 1104 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0212] Processor 1103 controls the operation of the network device. In specific applications, the various components of the network device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0213] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1103. Processor 1103 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1103. The above processor 1103 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1103 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1104, and processor 1103 reads information from memory 1104 and, in conjunction with its hardware, completes the steps of the above method.

[0214] Receiver 1101 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the network device. Transmitter 1102 can be used to output digital or character information through the first interface. Transmitter 1102 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group. Transmitter 1102 can also include a display device such as a display screen.

[0215] In an embodiment of the present application, in one case, the processor 1103 is used to cooperate with the terminal device to complete the collection of training data through the first model and the second model in the embodiment corresponding to Figure 4, etc.

[0216] An embodiment of the present application also relates to a computer storage medium, which stores a program for signal processing. When the program is run on a computer, it enables the computer to execute the steps executed by the aforementioned terminal device, or enables the computer to execute the steps executed by the aforementioned network device.

[0217] An embodiment of the present application also relates to a computer program product, which stores instructions that, when executed by a computer, enable the computer to execute the steps executed by the aforementioned terminal device, or enable the computer to execute the steps executed by the aforementioned network device.

[0218] An embodiment of the present application also provides a vehicle, which includes the aforementioned terminal device, exemplarily, the terminal device shown in Figure 8 or Figure 10.

[0219] The network device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0220] Specifically, see Figure 12, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 1200. NPU 1200 is mounted on the host CPU (host CPU) as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is the arithmetic circuit 1203, which is controlled by the controller 1204 to extract matrix data from the memory and perform multiplication operations.

[0221] In some implementations, the arithmetic circuit 1203 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1203 is a two-dimensional systolic array. The arithmetic circuit 1203 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1203 is a general-purpose matrix processor.

[0222] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1202 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1201 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1208.

[0223] Unified memory 1206 is used to store input and output data. Weight data is directly transferred to weight memory 1202 through the Direct Memory Access Controller (DMAC) 1205. Input data is also transferred to unified memory 1206 through the DMAC.

[0224] BIU stands for Bus Interface Unit, i.e., bus interface unit 1213 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1209 .

[0225] The bus interface unit 1213 (BIU) is used for the instruction fetch memory 1209 to obtain instructions from the external memory, and is also used for the storage unit access controller 1205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0226] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1206 or transfer weight data to the weight memory 1202 or transfer input data to the input memory 1201.

[0227] The vector calculation unit 1207 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1203, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of the predicted label plane.

[0228] In some implementations, the vector calculation unit 1207 can store the processed output vector to the unified memory 1206. For example, the vector calculation unit 1207 can apply a linear function or a nonlinear function to the output of the operation circuit 1203, such as linear interpolation of the predicted label plane extracted by the convolution layer, or another example is a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1207 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1203, for example, for use in subsequent layers in a neural network.

[0229] An instruction fetch buffer 1209 connected to the controller 1204 is used to store instructions used by the controller 1204;

[0230] Unified memory 1206, input memory 1201, weight memory 1202, and instruction fetch memory 1209 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0231] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0232] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0233] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0234] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0235] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A data collection method, characterized in that: The method comprises: The first terminal device receives a first feature from the network device, where the first feature indicates a first scenario; The first terminal device processes the second image to obtain a second feature, where the second feature indicates a second scene; The first terminal device determines, according to the first feature and the second feature, that if the first scene and the second scene belong to the same category of scenes, then the first terminal device sends the second image to the network device.

2. The method according to claim 1, characterized in that The first terminal device determines, according to the first feature and the second feature, that if the second image and the first scene belong to the same category of scenes, then the first terminal device sends the second image to the network device, including: If the first terminal device determines that the similarity between the first feature and the second feature is greater than or equal to a similarity threshold, the first terminal device sends the second image to the network device.

3. The method according to claim 1 or 2, characterized in that: The first terminal device receives the first feature from the network device, further comprising: The first terminal device receives a first parameter from the network device, where the first parameter is used to indicate a frame rate at which the first terminal subsequently sends images to the network device.

4. The method according to claim 3, characterized in that: The first terminal device sending the second image to the network device includes: The first terminal device sends the second image to the network device based on the first parameter.

5. The method according to any one of claims 1 to 4, characterized in that: After the method, it also includes: The first terminal device receives a second parameter from the network device, where the second parameter is used to adjust a frame rate at which the first terminal device sends an image to the network device; The first terminal device sends a third image to the network device based on the second parameter.

6. The method according to any one of claims 1 to 4, characterized in that: After the method, it also includes: The first terminal device receives a third parameter from the network device, where the third parameter is used to indicate a verification result of the second image by the network device, and sends a third image to the network device based on the verification result.

7. The method according to any one of claims 2 to 6, characterized in that: After the method, it also includes: The first terminal device receives a fourth parameter from the network device, and adjusts the similarity threshold based on the fourth parameter.

8. The method according to any one of claims 1 to 7, characterized in that: Prior to the method, it also includes: The second terminal device sends a first image to the network device based on a preset condition, where the first image belongs to the first scene.

9. The method according to claim 8, characterized in that The preset condition includes: the second terminal device performs a first operation for the first scenario, the second terminal device or the user performs a second operation for the first scenario, and the first operation and the second operation are different.

10. A data collection method, characterized in that: The method comprises: The network device sends a first feature to the first terminal device, where the first feature indicates a first scenario; The network device receives the second image sent by the first terminal device, and processes the second image to obtain a third feature, where the third feature indicates a second scene; The network device generates, based on the first feature and the third feature, a verification result of whether the second scene and the first scene belong to the same category of scenes; The network device sends a second parameter and / or a third parameter to the first terminal device, the second parameter is used to adjust the frame rate at which the first terminal device sends images to the network device, and the third parameter is used to indicate the verification result.

11. The method according to claim 10, characterized in that The network device sends the first feature to the first terminal device, further comprising: The network device sends a first parameter to the first terminal device, where the first parameter is used to indicate a frame rate at which the first terminal device subsequently sends an image to the network device.

12. The method according to claim 10 or 11, characterized in that: Also includes: The network device sends a fourth parameter to the first terminal device, where the fourth parameter is used to adjust a similarity threshold of the first terminal device.

13. The method according to any one of claims 10 to 12, characterized in that: Prior to the method, it also includes: The network device receives a first image sent by a second terminal device, and obtains the first feature based on the first image.

14. The method according to claim 13, characterized in that The obtaining the first feature based on the first image includes: Expanding the first image to obtain an image set, where the image set includes one or more images; The image set is processed to obtain the first feature.

15. A terminal device, characterized in that: The terminal device comprises: A receiving module, configured to receive a first feature from the network device, wherein the first feature indicates a first scenario; a processing module, configured to process the second image to obtain a second feature, wherein the second feature indicates a second scene; A sending module is used to determine, based on the first feature and the second feature, if the first scene and the second scene belong to the same category of scenes, and then send the second image to the network device.

16. A network device, characterized in that: The network equipment includes: A sending module, configured to send a first feature to a first terminal device, where the first feature indicates a first scenario; a receiving module, configured to receive a second image sent by the first terminal device, and process the second image to obtain a third feature, wherein the third feature indicates a second scene; A processing module, configured to generate a verification result of whether the second scene and the first scene belong to the same category of scenes based on the first feature and the third feature; The sending module is further used to send a second parameter and / or a third parameter to the first terminal, the second parameter is used to adjust the frame rate at which the first terminal sends images to the network device, and the third parameter is used to indicate the verification result.

17. A terminal device, characterized in that: The terminal device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the terminal device executes the method according to any one of claims 1 to 9.

18. A network device, characterized in that: The network device comprises a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the network device executes the method according to any one of claims 10 to 14.

19. A computer storage medium, characterized in that: The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to implement the method of any one of claims 1 to 14.

20. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 14.

21. A vehicle, characterized in that: The vehicle comprises the terminal device as claimed in claim 15 or 17.

Images