Data collection method and system, storage medium, electronic device and program product

Abstract

The present application provides a data acquisition method and system, a storage medium, an electronic device and a program product, and relates to the field of robotics. The method comprises: in response to a data acquisition task generation instruction for a target device, obtaining a target data acquisition task template from at least one data acquisition task template; obtaining configuration information for the target data acquisition task template, and generating an instance task corresponding to the data acquisition task generation instruction, so as to control the target device to execute the instance task. The present application enables data acquisition tasks to adapt to a variety of scenarios and requirements, and improves the pertinence and efficiency of data acquisition.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a data acquisition method and system, a storage medium, an electronic device, and a program product. Background Art

[0002] Currently, the embodied algorithm industry is committed to exploring end-to-end technology routes. This process requires a large amount of high-quality and diverse scene real machine collection data and simulation collection data to train the model and improve its generalization and success rate. Although this type of data set is large in scale, the modality of the perception data is not rich, which makes it difficult to meet the high-efficiency and targeted training requirements of embodied intelligent models in complex scenarios, thus limiting its wide application in downstream applications. Summary of the invention

[0003] In view of this, embodiments of the present application provide a data collection method and system, a storage medium, an electronic device, and a program product.

[0004] In a first aspect, an embodiment of the present application provides a data collection method, which is applied to an end-side device, and the method includes: in response to a data collection task generation instruction for a target device, obtaining a target data collection task template from at least one data collection task template; obtaining configuration information for the target data collection task template; and based on the configuration information, generating an instance task corresponding to the data collection task generation instruction, so as to control the target device to execute the instance task.

[0005] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: in response to a data collection instruction for a target device, presenting a data collection page, the data collection page displaying image data of the target device executing an instance task; in response to an upload operation triggered on the data collection page, uploading the perception data obtained by the target device executing the instance task to the cloud.

[0006] In combination with the first aspect, in certain implementation methods of the first aspect, before uploading the perception data obtained by the target device executing the instance task to the cloud, it also includes: performing validity verification on the perception data, and displaying the results of the validity verification on the data collection page, the validity verification includes an integrity check and / or a frame loss check; wherein, uploading the perception data obtained by the target device executing the instance task to the cloud includes: uploading the perception data that passes the validity verification to the cloud.

[0007] In combination with the first aspect, in certain implementations of the first aspect, the visualization processing includes data key frame annotation processing and / or data quality annotation processing, and the method also includes: in response to a data review and annotation instruction, obtaining data of a review and annotation page from the cloud, and presenting the review and annotation page, the review and annotation page including a data playback area and / or a review and annotation area; the review and annotation area includes at least one of an error cause label information box, a data frame selection box, and a text description box; receiving relevant information entered by the target user in the review and annotation area, and sending the information to the cloud.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: in response to a device registration instruction, presenting a device registration page, the device registration page including at least one of a main body type information box, a terminal type information box, and a sensor type information box, the main body type information box, the terminal type information box, and the sensor type information box respectively corresponding to multiple options; receiving relevant information entered by the target user on the device registration page to complete the device registration.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the data collection task template includes scene text generalization configuration items and collection action configuration items; the scene text generalization configuration items include at least one configuration item of action type, state change time, and scene initial state; the collection action configuration items include at least one configuration item of action steps, atomic capabilities, and operation objects.

[0010] In the second aspect, an embodiment of the present application provides a data collection method, which is applied to the cloud; the method includes: acquiring perception data of a target device, the perception data of the target device is obtained by the target device executing an instance task generated by the first aspect; processing the perception data of the target device, and the processing includes at least one of timestamp alignment processing, data compression processing and visualization processing.

[0011] In combination with the second aspect, in certain implementations of the second aspect, the visualization processing includes data key frame annotation processing and / or data quality annotation processing, and processes the perception data of the target device, including: sending the data of the review annotation page to the end-side device; obtaining relevant information sent by the end-side device, and performing data key frame annotation processing and / or data quality annotation processing on the perception data based on the information, wherein the information is information entered by the target user in the review annotation page presented by the end-side device, and the review annotation page includes a data playback area and / or a review annotation area; the review annotation area includes at least one of an error cause label information box, a data frame selection box, and a text description box.

[0012] On the third aspect, an embodiment of the present application provides a data acquisition system, including: a cloud, storing at least one data acquisition task template; an end-side device, which is communicatively connected to the cloud, and the end-side device is configured to respond to a data acquisition task generation instruction for a target device, obtain a target data acquisition task template from at least one data acquisition task template; obtain configuration information for the target data acquisition task template; and generate an instance task corresponding to the data acquisition task generation instruction based on the configuration information, so as to control the target device to execute the instance task.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the data collection method described in the first aspect and the second aspect.

[0014] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; the processor is used to execute the data collection method described in the first aspect and the second aspect.

[0015] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on an electronic device, the electronic device implements the data collection method described in the first aspect and the second aspect.

[0016] In the present application, first, the target data acquisition task template obtained provides a standardized starting point for subsequent data acquisition tasks. In addition, the present embodiment allows personalized configuration of the target data acquisition task template. Such flexibility enables the data acquisition task to adapt to a variety of scenarios and needs, thereby enriching the modality of the perceived data. At the same time, the configuration information provides parameter guidance for generating specific instance tasks, ensuring that the data acquisition task can be accurately executed according to specific requirements, thereby improving the pertinence and efficiency of data acquisition. In short, the scheme in the present embodiment makes the data acquisition process more flexible and efficient, and can generate richer and higher-quality data sets to support the training and application of embodied intelligence models, thereby broadening the extensiveness of embodied intelligence models in downstream applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1The figure is a flow chart of a data collection method provided in one embodiment of the present application.

[0019] Figure 2 Shown is a schematic diagram of a basic information configuration page provided in an embodiment of the present application.

[0020] Figure 3 Shown is a schematic diagram of an action template configuration page provided in an embodiment of the present application.

[0021] Figure 4 Shown is a flow chart of a data collection method provided in another embodiment of the present application.

[0022] Figure 5 Shown is a schematic diagram of a page before data collection starts provided by an embodiment of the present application.

[0023] Figure 6a Shown is a schematic diagram of a page in data collection provided by an embodiment of the present application.

[0024] Figure 6b Shown is a schematic diagram of a page in data collection provided by another embodiment of the present application.

[0025] Figure 7 The figure is a flow chart of a data collection method provided by another embodiment of the present application.

[0026] Figure 8 Shown is a schematic diagram of an audit annotation page provided in an embodiment of the present application.

[0027] Fig. 9 Shown is a flow chart of a data collection method provided in yet another embodiment of the present application.

[0028] Fig.10 A schematic diagram of a device registration page provided in accordance with an embodiment of the present application.

[0029] Fig.11 The figure is a flow chart of a data collection method provided in one embodiment of the present application.

[0030] Fig.12 Shown is a schematic diagram of the structure of a data acquisition system provided in one embodiment of the present application.

[0031] Fig.13 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] Figure 1 FIG. 1 is a flow chart of a data collection method provided by an embodiment of the present application. The method is applied to a terminal device, and exemplarily, the terminal device is a computer. Figure 1 As shown, the method includes the following steps.

[0034] Step S110 , in response to a data collection task generation instruction for a target device, obtaining a target data collection task template from at least one data collection task template.

[0035] Optionally, at least one data collection task template is stored in the cloud, and thus can be accessed by authorized users from any location, increasing flexibility and convenience. In addition, the cloud provides unlimited storage space, which can be easily expanded to store more data collection task templates.

[0036] Optionally, the data collection task generation instruction is generated when the target user triggers a specific operation in the data collection client deployed on the terminal side. The data collection task template is a set of preset parameters used to guide the data collection process. The target data collection task template is selected from at least one data collection task template.

[0037] Optionally, the target device is an intelligent robot.

[0038] Step S120, obtaining configuration information of the target data collection task template.

[0039] In one example, the terminal-side device may display the target data collection task template to the target user, with the purpose of allowing the target user to personalize the target data collection task template according to specific needs, thereby obtaining configuration information of the target data collection task template.

[0040] Specifically, the configuration information refers to the specific parameters entered or selected when personalizing the target data collection task template, including but not limited to the type of data collected, collection frequency, collection duration, specific configuration of the sensor, collection conditions, parameter thresholds, etc., to ensure that the data collection task can be performed according to specific requirements.

[0041] Step S130: Generate an instance task corresponding to the data acquisition task generation instruction based on the configuration information.

[0042] Optionally, based on the configuration information, a configuration file containing all necessary instructions and parameters is created, which describes in detail the steps and conditions that the target device needs to perform. Exemplarily, after the configuration file is created, the target user (such as a collector) uploads it to the control system of the target device through a wired connection, a wireless network or other communication methods. The control system of the target device reads the configuration file and performs corresponding actions according to the instructions therein.

[0043] In this embodiment, first, the target data acquisition task template obtained provides a standardized starting point for subsequent data acquisition tasks. In addition, this embodiment allows personalized configuration of the target data acquisition task template. Such flexibility enables the data acquisition task to adapt to a variety of scenarios and needs, thereby enriching the modality of the perceived data. At the same time, the configuration information provides parameter guidance for generating specific instance tasks, ensuring that the data acquisition task can be accurately executed in accordance with specific requirements, thereby improving the pertinence and efficiency of data acquisition. In short, the scheme in this embodiment makes the data acquisition process more flexible and efficient, and can generate richer and higher-quality data sets to support the training and application of embodied intelligence models, thereby broadening the breadth of embodied intelligence models in downstream applications.

[0044] exist Figure 1 In the corresponding embodiment, a preliminary overview of the data collection task template is given. Figure 2 and Figure 3 , further explore the details of the data collection task template. Specifically, the data collection task template is created through the basic information configuration page and the action template configuration page, where Figure 2 FIG. 1 is a schematic diagram of a basic information configuration page provided by an embodiment of the present application. Figure 3 Shown is a schematic diagram of an action template configuration page provided in an embodiment of the present application.

[0045] like Figure 2 As shown, the basic information configuration page allows the target user to enter the necessary information for a specific data collection task. Optionally, the page includes:

[0046] Task name: used to customize a name for the data collection task template;

[0047] Task purpose: including but not limited to formal, trial, test and simulation evaluation, which helps the target user to carry out corresponding processing and configuration according to the nature of the data collection task;

[0048] Scene labels: such as industrial scenes and home scenes, which help to classify and retrieve data collection tasks;

[0049] Collection ontology: used to specify the subject of data collection, such as a selectable or input robot or other device;

[0050] End effector type: For example, you can choose the robot's arm, tool, or other end effector;

[0051] Remote control type: including but not limited to VR (virtual reality) and motion capture (motion capture);

[0052] Collection mode: including but not limited to the whole body, both arms, left arm and right arm, which determines the coverage of data collection;

[0053] Sensor type: For example, you can select or input a visual sensor, a tactile sensor, or a lidar sensor;

[0054] Task Notes: Used to add additional notes to provide more details about the data collection task.

[0055] like Figure 3 As shown, the action template configuration page allows the target user to define specific action steps and related parameters in the data collection task. Exemplarily, the page includes scene text generalization configuration items and collection action configuration items. Among them, the scene text generalization configuration item involves the generalization ability of the scene description in the data collection task, which mainly refers to the conversion of the specific environment description into a more general and broader expression, so that the data collection task template can adapt to a variety of different environments and conditions. The collection action configuration item involves the setting of specific actions in the data collection task, mainly defining the specific actions or behaviors that the device needs to perform during the data collection process.

[0056] Furthermore, the scene text generalization configuration items include:

[0057] Action type: including but not limited to easy, medium and difficult, in order to classify the difficulty of the action;

[0058] State change time: used to specify the update frequency of the scene initial state and action text during the data collection process. For example, the scene initial state and action text will be updated after each collection is completed.

[0059] Scene initial state: used to define the scene state before the action starts. For example, the scene initial state is set to the kettle being 30m away from the robot and half full of water.

[0060] The collection action configuration items include:

[0061] Action steps: including but not limited to duration (s), deviation (s), action text, atomic ability, and operation; among them, duration (s) is used to set the duration of each action step; deviation (s) is used to set the deviation range of the allowed action execution; action text is used to describe the action step, for example, the action text is to pick up the kettle 30m away from the table; atomic ability is used to set the basic skills involved in the action step, such as grabbing, moving, etc.; operation is used to edit or delete each action step;

[0062] Manipulating objects: used to input or select object types, such as fruit and furniture;

[0063] Action duration: used to define the total duration of the entire action step;

[0064] Countdown: used to set the countdown time before the action step is executed;

[0065] Unusual proportion: It is used to define the proportion of unconventional or abnormal actions in the execution of action steps in order to simulate the uncertainty in the real world.

[0066] It should be noted that in the collection action configuration item, it can be a single action or a combination of a series of actions, that is, a multi-step configuration. For example, if you need to collect data when the robot performs a series of specific tasks (such as grasping, moving, and placing objects), the multi-step configuration can define each action of these tasks in detail to ensure the accuracy and consistency of data collection. Figure 3 As shown, two action steps are configured.

[0067] In this embodiment, through the basic information configuration page, the target user can set parameters and goals for the data collection task, which helps to ensure the purpose of the data collection task. The action template configuration page allows the target user to define the specific action steps in the data collection process, which increases the accuracy and repeatability of the data collection task. The combination of these two pages makes the data collection task more flexible and efficient, able to adapt to a variety of scenarios and needs, while improving the quality and applicability of data collection, and providing a basis for subsequent data analysis and model training.

[0068] Figure 4 FIG. 1 is a flow chart of a data collection method provided by another embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 4 The embodiment shown is described below in detail. Figure 4 The embodiment shown and Figure 1 The differences and similarities of the illustrated embodiments are not described in detail.

[0069] like Figure 4As shown, the method also includes the following steps.

[0070] Step S410: presenting a data collection page in response to a data collection instruction for a target device.

[0071] The data collection instruction refers to the command received from the target user, which is used to instruct the target device to start executing a specific data collection task and start the data collection process. At the same time, the image data of the target device executing the instance task is displayed through the data collection page.

[0072] Optionally, the data collection page includes a page before data collection starts and / or a page during data collection. Figure 5 Shown is a schematic diagram of a page before data collection starts provided by an embodiment of the present application. Figure 6a Shown is a schematic diagram of a page in data collection provided by an embodiment of the present application.

[0073] like Figure 5 As shown in the figure, the page before data collection starts provides an overview and control options of the data collection task. This page mainly includes:

[0074] Scene initial state: used to display the scene settings before data collection begins;

[0075] Action duration: used to display the estimated duration of the entire data collection action, such as 38-40s;

[0076] Collection progress: used to display the progress of data collection, including but not limited to the percentage completed;

[0077] Unusual data: used to display unusual or unusual data points encountered during the data collection process;

[0078] Action Steps: used to display the specific action steps in the data collection process. The action steps refer to the action steps pre-configured in the target data collection task template.

[0079] Image data display area: used to display in real time the image data captured by the target device when executing the instance task, so as to monitor the real-time progress of data collection. The image data includes video frames;

[0080] Exit Collection: A button that allows the target user to terminate or exit the current data collection task;

[0081] Historical data: button for viewing previously collected data records;

[0082] Start Collection: Button used to initiate the data collection process.

[0083] like Figure 6aAs shown in the figure, the data collection page provides detailed information and control options about the data collection status. The page mainly includes:

[0084] File duration: used to display the duration of the data file, such as 55 seconds, to help the target user understand the duration of data collection;

[0085] File size: used to display the size of the data file, such as 315.2MB;

[0086] Data type: including but not limited to conventional and unconventional, which helps the target user to classify data files;

[0087] Image data display area: also used to display the image data captured by the target device when executing the instance task in real time, so as to monitor the real-time progress of data collection;

[0088] Abandon and recollect: A button that allows the target user to abandon the currently collected data and restart the collection process;

[0089] Submit and upload: If there are no problems during the data collection process, the target user can click this button to upload the data.

[0090] from Figure 5 and Figure 6a It can be seen that the page before data collection starts provides a comprehensive understanding of the data collection task that is about to begin, and the page during data collection provides users with a comprehensive understanding of the ongoing data collection task. At the same time, both provide options for controlling and monitoring the data collection process. Through this page, the target user can ensure that the data collection task is proceeding as expected and intervene when necessary.

[0091] Step S420, in response to the upload operation triggered on the data collection page, the perception data obtained by the target device executing the instance task is uploaded to the cloud.

[0092] Exemplarily, when the target user clicks the "Submit Upload" button, it is considered that the upload operation is triggered on the data collection page, and at this time, the perception data obtained when the target device performs the instance task is uploaded to the cloud so that the cloud can process the perception data. Optionally, the processing includes at least one of timestamp alignment processing, data compression processing, and visualization processing.

[0093] Perception data refers to data directly obtained by the target device from the environment through its sensors during the data collection process, which reflects the information perceived by the target device when performing the instance task. Optionally, the perception data includes image data (including video data), tactile data, position information, speed, etc.

[0094] In addition, during the data collection process, different sensors or data channels will record data independently, and each of these data has its own timestamp. Due to the possible differences in the sampling rate, startup time or processing delay of the sensor, the data of different channels may not correspond precisely in time, that is, their timestamps may be inconsistent.

[0095] Timestamp alignment processing can solve this problem and ensure that the data of all channels can be synchronized according to a unified time reference. In this way, when analyzing the perception data, it can be ensured that the data from different sensors occurs at the same time point or time period, thereby improving the consistency and comparability of the perception data. Specifically, during the timestamp alignment process, a main channel needs to be selected as a reference point for time synchronization, and then the data of other channels will be adjusted according to the timestamp of this main channel. Optionally, the adjustment process involves frame extraction (removing redundant frames to match the time point of the main channel) and frame filling (inserting additional frames to fill the time difference). In this way, it can be ensured that the data of all channels are aligned in time.

[0096] During the data acquisition process, images captured by the camera are usually stored in formats such as JPEG or PNG. Although these formats have high quality, the size of each file is relatively large. When a large number of image files are generated by the same data acquisition task, these scattered small files not only take up a lot of storage space, but may also be difficult to manage due to their large number. To solve this problem, these image files can be compressed into video formats such as MP4, H.264, or H.265 after timestamp alignment processing to remove redundant information and use the similarity between frames to reduce the amount of data.

[0097] Visualization processing refers to converting perception data into graphics or images so that the target users can intuitively understand the information represented by the perception data, thereby helping the target users to view anomalies in the perception data.

[0098] In this embodiment, by presenting the data collection page, the target user can monitor the image data of the target device executing the instance task in real time, which increases transparency and allows the target user to monitor and intervene in the data collection process in real time. When the target user triggers the upload operation, the perception data is uploaded to the cloud, and the powerful processing capabilities of the cloud are used to further process the perception data. These processing steps together improve the quality and availability of the perception data and improve the overall efficiency of data collection.

[0099] In another implementation of the present application, before step S420, it also includes: performing validity verification on the perception data, and displaying the results of the validity verification on the data collection page, the validity verification includes integrity verification and / or frame loss verification. Further, step S420 includes: uploading the perception data that passes the validity verification to the cloud. Alternatively, the data format of the perception data that passes the validity verification is unified, and then uploaded to the cloud.

[0100] Figure 6b The figure shows a schematic diagram of a page in data collection provided by another embodiment of the present application. Figure 6a Based on this, the pages in data collection also include:

[0101] File integrity check: used to show whether the data file corresponding to the perception data has passed the integrity check. If the check passes, it means that the data file has not been damaged or key frame information has been lost during the acquisition process;

[0102] Camera frame number / frame rate verification: used to show whether the frame number and frame rate captured by the camera meet expectations. If the verification fails, it means that there are problems in the data acquisition process, such as unstable frame rate or frame loss.

[0103] In this embodiment, before uploading the perception data to the cloud, a validity verification step is added to ensure that only perception data with qualified quality and accuracy will be uploaded and further processed, thereby improving the reliability of the data and the accuracy of subsequent analysis. Validity verification includes integrity verification and frame loss verification. Through these verifications, problems that may arise during the data collection process can be discovered and eliminated in a timely manner. The results of the validity verification are displayed on the data collection page, providing intuitive feedback to the target user, so that the target user can understand the quality status of the data and take corresponding measures when necessary, such as discarding and re-collecting. Only perception data that has passed the validity verification will be uploaded to the cloud, which optimizes the efficiency of data transmission and ensures that the data processed by the cloud is accurate and reliable.

[0104] Figure 7 FIG. 1 is a flow chart of a data collection method provided by another embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 7 The embodiment shown is described below in detail. Figure 7 The embodiment shown and Figure 1 The differences and similarities of the illustrated embodiments are not described in detail.

[0105] like Figure 7 As shown, the method also includes the following steps.

[0106] Step S710, in response to the data review and annotation instruction, obtain the data of the review and annotation page from the cloud, and present the review and annotation page.

[0107] Exemplarily, the data review and annotation instruction is triggered by the target user to start the data review and annotation process. In addition, the review and annotation software is running on the cloud server and provides the review and annotation function. The end-side device is connected to the cloud via the Internet. After the target user issues the data review and annotation instruction, the data collection client on the end-side device requests the review and annotation page on the cloud. The cloud sends the data of the review and annotation page to the end-side device, which further renders the data and finally displays the review and annotation page.

[0108] Figure 8 FIG. 1 is a schematic diagram of an audit markup page provided by an embodiment of the present application. Figure 8 As shown, the review and annotation page includes a data playback area and / or a review and annotation area; the review and annotation area includes at least one of an error cause label information box, a data frame selection box and a text description box.

[0109] Specifically, the data playback area is used to display perception data, such as video or image sequences, where the target user can view the data and review and annotate them. The error cause label is an input box for entering the specific reason for the error during data collection in order to identify and solve the problem.

[0110] The data frame selection box includes the region frame selection box and the single frame selection box. The region frame selection box is used to mark the start frame and end frame of a specific region in the perception data. The target user can enter the frame number in the region frame selection box and add description text in the text description box to mark the specific part of the perception data in detail. The single frame selection box allows the target user to mark a single frame, which also includes entering the frame number and description text. It should be noted that the number of region frame selection boxes and single frame selection boxes can be increased or decreased according to demand.

[0111] In addition to the above, the audit mark areas also include:

[0112] Data type: used to display or allow the target user to select the type of perceived data, such as unconventional;

[0113] Data validity: including but not limited to valid and invalid, which helps to filter out data that does not meet the requirements;

[0114] Action completion: including but not limited to completion and incomplete, so as to evaluate the execution of instance tasks;

[0115] Submit and Next: After completing the review and annotation of the current data, the target user can click this button to submit the current work and move to the next data;

[0116] Submit and return: After completing the review and annotation, the user submits the current work and returns to the main interface or the previous menu.

[0117] like Figure 8 As shown in the figure, the review and annotation page not only provides the target user with the option of detailed inspection and annotation of the collected data, but also provides an intuitive interface through which the target user can effectively manage and control the quality of data collection (including the elimination of abnormal data), providing a higher quality data foundation for subsequent data analysis and model training.

[0118] Step S720, receiving relevant information input by the target user in the review mark area, and sending the information to the cloud.

[0119] Specifically, the target user performs annotation and review operations on the end-side device through the review and annotation page, and these operations will be sent to the cloud so that the cloud can perform key frame annotation processing and / or data quality annotation processing on the perception data based on these operations. Among them, key frame annotation processing involves identifying and marking key events or change points in the perception data, while data quality annotation processing includes evaluating the overall quality of the perception data and marking errors or anomalies in the perception data. These processing steps help to improve the availability of perception data, making the perception data more suitable for subsequent analysis, model training or other applications.

[0120] In this embodiment, by presenting the review and annotation page, the target user can directly observe the perceived data, conduct detailed inspection and annotation, so as to capture the key information in the data. By receiving the relevant information entered by the target user in the review and annotation area and sending this information to the cloud, the cloud can be used to perform actual key frame annotation and data quality annotation, which fully utilizes the resources and advantages of the cloud, improves the efficiency and quality of data processing, and reduces the burden and cost of the terminal side equipment.

[0121] Fig. 9 FIG. 1 is a flow chart of a data collection method provided by another embodiment of the present application. Figure 1 Based on the embodiment shown, Fig. 9 The embodiment shown is described below in detail. Fig. 9 The embodiment shown and Figure 1 The differences and similarities of the illustrated embodiments are not described in detail.

[0122] like Fig. 9 As shown, the method also includes the following steps.

[0123] Step S910: In response to the device registration instruction, present a device registration page.

[0124] For example, the device registration instruction is issued by the target user for the purpose of registering a new device. In response, the terminal device will present a device registration page, in which the user can enter or confirm the relevant information of the new device. Fig.10 This is a schematic diagram of a device registration page provided in an embodiment of the present application. Fig.10 As shown, the device registration page includes:

[0125] Device name: Users can enter the name of the device here for easy identification and management;

[0126] AID: refers to the asset identifier (Asset ID), which is used to uniquely identify each device;

[0127] Body type: refers to the main part of the equipment, such as single-arm robot, dual-arm robot, etc.

[0128] End type: such as end of arm or other end effector;

[0129] Sensor type: such as visual sensor, tactile sensor, lidar, etc.;

[0130] Access status: used to set whether the device is allowed to access the data acquisition system or network;

[0131] Search and reset buttons: Allow the target user to search for a specific device or reset the search criteria;

[0132] Registered devices: used to display the number of registered devices;

[0133] Remaining number of devices that can be registered: used to display the number of new devices that can be registered;

[0134] After the device is registered, detailed information of the registered device will be displayed below, including ID, device name, AID, body type, terminal type, sensor type, IP address, registration time, active time, access status and operation options. The operation options can further select encoding and decoding, downloading logs, deletion, etc.

[0135] like Fig.10 As shown in Figure 1, the device registration page allows users to register and manage devices participating in data collection. Through this page, users can ensure that all devices are correctly identified and configured so that they can participate in data collection tasks in the intended manner.

[0136] Step S920, receiving relevant information input by the target user on the device registration page to complete the device registration.

[0137] In this embodiment, through the device registration page, multi-entity, multi-terminal, and multi-modal device registration can be completed, providing a basis for device management and monitoring. It also lays the foundation for subsequent data collection and processing.

[0138] Fig.11 FIG. 1 is a flow chart of a data collection method provided by an embodiment of the present application. Exemplarily, the method is applied to a cloud storing at least one data collection task template. Fig.11 As shown, the method includes the following steps.

[0139] Step S110, acquiring the perception data of the target device.

[0140] Specifically, the perception data refers to the data directly obtained by the target device from the environment through its sensors during the data collection process, which reflects the information perceived by the target device when performing the instance task. Optionally, the perception data includes image data (including video data), tactile data, position information, speed, etc.

[0141] Step S1120: Process the perception data of the target device.

[0142] The processing includes at least one of a timestamp alignment process, a data compression process, and a visualization process.

[0143] Specifically, the timestamp alignment process ensures that the data from different sensors or data channels can be synchronized according to a unified time reference. In this way, when analyzing the perception data, it can be ensured that the data from different sensors occurs at the same time point or time period, thereby improving the consistency and comparability of the perception data. Specifically, during the timestamp alignment process, a main channel needs to be selected as a reference point for time synchronization, and then the data of other channels will be adjusted according to the timestamp of this main channel. Optionally, the adjustment process involves frame extraction (removing redundant frames to match the time point of the main channel) and frame filling (inserting additional frames to fill the time difference). In this way, it can be ensured that the data of all channels are aligned in time.

[0144] During the data acquisition process, images captured by the camera are usually stored in formats such as JPEG or PNG. Although these formats have high quality, the size of each file is relatively large. When a large number of image files are generated by the same data acquisition task, these scattered small files not only take up a lot of storage space, but may also be difficult to manage due to their large number. To solve this problem, these image files can be compressed into video formats such as MP4, H.264, or H.265 after timestamp alignment processing to remove redundant information and use the similarity between frames to reduce the amount of data.

[0145] Visualization processing refers to converting perception data into graphics or images so that the target users can intuitively understand the information represented by the perception data, thereby helping the target users to view anomalies in the perception data.

[0146] In this embodiment, the powerful processing capabilities of the cloud are used to further process the perception data, such as time stamp alignment to ensure data synchronization, data compression to improve storage and transmission efficiency, and visualization to make data easier to understand and analyze. These processing steps together improve the quality and availability of perception data and improve the overall efficiency of data collection.

[0147] Combination Fig.11 As shown, in some other embodiments of the present application, the visualization processing includes data key frame annotation processing and / or data quality annotation processing, and the perception data of the target device is processed, including: sending the data of the review annotation page to the terminal device; obtaining the relevant information sent by the terminal device, and performing data key frame annotation processing and / or data quality annotation processing on the perception data based on the information, wherein the information is the information entered by the target user in the review annotation page presented by the terminal device, and the review annotation page includes a data playback area and / or a review annotation area; the review annotation area includes at least one of an error cause label information box, a data frame selection box and a text description box.

[0148] As mentioned above, the audit and annotation software is running on the cloud server, and provides the audit and annotation function. The end-side device is connected to the cloud via the Internet. After the target user issues a data audit and annotation instruction, the data collection client on the end-side device requests the audit and annotation page on the cloud. At this time, the cloud sends the data of the audit and annotation page to the end-side device, and the end-side device further renders the data and finally displays the audit and annotation page. The target user performs annotation and audit operations on the audit and annotation page on the end-side device. These operations (i.e., relevant information) will be sent to the cloud, and the cloud will perform key frame annotation processing and / or data quality annotation processing on the perceived data based on these operations.

[0149] Specifically, keyframe annotation involves identifying and marking key events or change points in the perception data, while data quality annotation includes evaluating the overall quality of the perception data and marking errors or anomalies in the perception data. These processing steps help improve the usability of perception data, making it more suitable for subsequent analysis, model training or other applications.

[0150] In this embodiment, the cloud is used to perform actual key frame labeling and data quality labeling, which fully utilizes the resources and advantages of the cloud, improves the efficiency and quality of data processing, and reduces the burden and cost of the terminal-side equipment.

[0151] Fig.12 FIG. 1 is a schematic diagram of the structure of a data acquisition system provided by an embodiment of the present application. Fig.12 As shown, the data acquisition system includes a cloud 1210 and an end-side device 1220, and the cloud 1210 and the end-side device 1220 are communicatively connected.

[0152] The cloud 1210 stores at least one data collection task template.

[0153] Specifically, a data collection task template is a set of preset parameters used to guide the data collection process.

[0154] The terminal side device 1220 is configured to obtain a target data acquisition task template from at least one data acquisition task template in response to a data acquisition task generation instruction for a target device; obtain configuration information for the target data acquisition task template; and generate an instance task corresponding to the data acquisition task generation instruction based on the configuration information, so as to control the target device to execute the instance task.

[0155] Exemplarily, the data collection task generation instruction is generated when the target user triggers a specific operation in the data collection client deployed on the end side. The target data collection task template is selected from at least one data collection task template. Configuration information refers to specific parameters entered or selected when personalizing the target data collection task template, including but not limited to the type of data collected, collection frequency, collection duration, specific configuration of the sensor, collection conditions, parameter thresholds, etc., to ensure that the data collection task can be executed according to specific requirements.

[0156] Furthermore, based on the configuration information, a configuration file containing all necessary instructions and parameters is created. This configuration file describes in detail the steps and conditions that the target device needs to perform. After the configuration file is created, the target user (such as a collector) uploads it to the control system of the target device through a wired connection, a wireless network, or other communication methods. The control system of the target device reads the configuration file and performs corresponding actions according to the instructions therein.

[0157] In this embodiment, first, the target data collection task template obtained by the end-side device provides a standardized starting point for subsequent data collection tasks. In addition, this embodiment allows personalized configuration of the target data collection task template. Such flexibility enables the data collection task to adapt to a variety of scenarios and needs, thereby enriching the modality of the perceived data. At the same time, the configuration information provides parameter guidance for generating specific instance tasks, ensuring that the data collection tasks can be accurately performed in accordance with specific requirements, thereby improving the pertinence and efficiency of data collection. In short, the data collection system in this embodiment makes the data collection process more flexible and efficient, and can generate richer and higher-quality data sets to support the training and application of embodied intelligence models, thereby broadening the breadth of embodied intelligence models in downstream applications.

[0158] In addition, it can be seen from the comprehensive method embodiments that the data acquisition system provided by this application can simultaneously meet the data acquisition requirements of multiple entities (such as different types of robots), multiple terminals (such as different operating parts of robots), multiple modalities (such as multiple types of data input) and unified formats. Moreover, combined with the cloud, real-time data collection, processing and management are realized, the efficiency and quality of data collection are improved, and the complexity and cost of data processing are reduced, which can better meet the needs of the embodied intelligence industry.

[0159] It should be understood that the description of the data acquisition method embodiment corresponds to the description of the data acquisition system embodiment, and therefore, for the parts of the data acquisition system that are not described in detail, reference can be made to the previous method embodiment.

[0160] Below, reference Fig.13 To describe an electronic device according to an embodiment of the present application. Fig.13 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0161] like Fig.13 As shown, the electronic device 130 includes one or more processors 1301 and a memory 1302 .

[0162] The processor 1301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 130 to perform desired functions.

[0163] The memory 1302 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1301 may run the program instructions to implement the data acquisition method of each embodiment of the present application described above and / or other desired functions. Various contents such as data acquisition task templates, configuration information, and instance tasks may also be stored in the computer-readable storage medium.

[0164] In one example, the electronic device 130 may further include: an input device 1303 and an output device 1304 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0165] The input device 1303 may include, for example, a keyboard, a mouse, etc.

[0166] The output device 1304 can output various information to the outside, including data collection task templates, configuration information, instance tasks, etc. The output device 1304 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0167] Of course, to simplify, Fig.13 Only some of the components related to the present application in the electronic device 130 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 130 may also include any other appropriate components.

[0168] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the data acquisition method according to various embodiments of the present application described above in this specification.

[0169] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0170] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the data acquisition method according to various embodiments of the present application described above in this specification.

[0171] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0172] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0173] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0174] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0175] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0176] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A data collection method, characterized in that: Applied to a terminal device, the method includes: In response to a data collection task generation instruction for a target device, acquiring a target data collection task template from at least one data collection task template; Acquire configuration information of the target data acquisition task template; Based on the configuration information, an instance task corresponding to the data acquisition task generation instruction is generated to control the target device to execute the instance task.

2. The data collection method according to claim 1, characterized in that: Also includes: In response to the data collection instruction for the target device, presenting a data collection page, wherein the data collection page displays image data of the target device executing the instance task; In response to an upload operation triggered on the data collection page, the perception data obtained by the target device executing the instance task is uploaded to the cloud.

3. The data collection method according to claim 2, characterized in that: Before uploading the perception data obtained by the target device executing the instance task to the cloud, the method further includes: Performing validity check on the perception data and displaying the result of the validity check on the data collection page, wherein the validity check includes integrity check and / or frame loss check; The uploading of the perception data obtained by the target device executing the instance task to the cloud includes: uploading the perception data that passes the validity check to the cloud.

4. The data collection method according to claim 1, characterized in that: Also includes: In response to the data review and annotation instruction, data of the review and annotation page is obtained from the cloud, and the review and annotation page is presented, wherein the review and annotation page includes a data playback area and / or a review and annotation area; the review and annotation area includes at least one of an error cause label information box, a data frame selection box, and a text description box; Receive the relevant information entered by the target user in the review mark area, and send the information to the cloud.

5. The data collection method according to any one of claims 1 to 4, characterized in that: Also includes: In response to the device registration instruction, presenting a device registration page, the device registration page including at least one of a body type information box, a terminal type information box, and a sensor type information box, wherein the body type information box, the terminal type information box, and the sensor type information box respectively correspond to a plurality of options; Receive relevant information entered by the target user on the device registration page to complete device registration.

6. The data collection method according to any one of claims 1 to 4, characterized in that: The data collection task template includes a scene text generalization configuration item and a collection action configuration item; the scene text generalization configuration item includes at least one configuration item of action type, state change time, and scene initial state; the collection action configuration item includes at least one configuration item of action steps, atomic capabilities, and operation objects.

7. A data processing method, characterized in that: Applied in the cloud; the method comprises: Acquire perception data of a target device, wherein the perception data of the target device is obtained by the target device executing an instance task generated by any one of claims 1 to 6; The perception data of the target device is processed, and the processing includes at least one of timestamp alignment processing, data compression processing and visualization processing.

8. The data processing method according to claim 7, characterized in that: The visualization processing includes data key frame annotation processing and / or data quality annotation processing, and the processing of the perception data of the target device includes: Send the data of the review and annotation page to the end-side device; Obtain relevant information sent by the end-side device, and perform the data key frame annotation processing and / or the data quality annotation processing on the perception data based on the information, wherein the information is information entered by the target user in the review annotation page presented by the end-side device, and the review annotation page includes a data playback area and / or a review annotation area; the review annotation area includes at least one of an error cause label information box, a data frame selection box and a text description box.

9. A data acquisition system, characterized in that: include: The cloud stores at least one data collection task template; A terminal device is connected to the cloud for communication, and the terminal device is configured to obtain a target data collection task template from at least one data collection task template in response to a data collection task generation instruction for a target device; and obtain configuration information for the target data collection task template; Based on the configuration information, an instance task corresponding to the data acquisition task generation instruction is generated to control the target device to execute the instance task.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the data collection method according to any one of claims 1 to 8 above.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the data collection method described in any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises instructions, which, when executed on an electronic device, enable the electronic device to implement the data collection method according to any one of claims 1 to 8.

Images