A method and device for reproducing equipment operation process

Abstract

This specification discloses a method and device for reproducing the operation process of a device, which can be used for unmanned driving devices. The method includes: first receiving a reproduction request that at least includes a reproduction time period, and then obtaining the status information of each module in the device collected during the reproduction time period and the event data corresponding to each module according to the reproduction request, and determining the initial state of each module according to the status information of each module, building an initial simulation environment, and determining the triggering time of each event according to the event data corresponding to each module. Finally, according to the triggering time of each event and the event data corresponding to each module, the events executed by each module during the operation of the device are reproduced in the initial simulation environment. Since the initial state of each module and the triggering time of each event are kept consistent with the actual operation process during the simulation reproduction process, the reproduction process is highly consistent with the actual operation process, and the operation process of the device can be highly reproduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and apparatus for reproducing the operation process of a device. Background Art

[0002] At present, with the development of computer technology, the debugging technology of equipment is also constantly improving. In order to troubleshoot problems that occur during the operation of equipment in the actual environment, simulation is usually used to reproduce the operation process of the equipment.

[0003] In the prior art, in order to reproduce the operation process of a device, it is generally necessary to determine the execution order of each event in the device during the period when the problem occurs, so as to reproduce the events in the simulation system in the order in which they occur, simulate the operation process of the device in the actual environment when the problem occurs, and finally determine the cause of the problem based on the execution process and results.

[0004] However, the current simulation system only reproduces the device operation process in the order of events, which makes it difficult to restore the complexity of the actual scene, making it difficult to determine the cause of the problem. It can be seen that there is an urgent need for a method that can highly reproduce the operation process of the device in the actual environment, so that the cause of the problem can be determined based on the reproduced execution process and execution results. Summary of the invention

[0005] The embodiments of this specification provide a method and apparatus for reproducing the operation process of a device, which are used to at least partially solve the problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This manual provides a method for reproducing the operation process of a device, including:

[0008] receiving a recurrence request, wherein the recurrence request at least includes a recurrence time period;

[0009] According to the reproduction request, the state information of each module in the device collected during the reproduction period and the event data corresponding to each module are obtained;

[0010] According to the status information of each module, determine the initial status of each module of the device and build an initial simulation environment;

[0011] Determine the triggering time of each event according to the event data corresponding to each module;

[0012] According to the triggering time of each event and the event data corresponding to each module, the events executed by each module during the operation of the device are reproduced in the initial simulation environment.

[0013] Optionally, the reproduction request further includes module identifiers of at least some modules in the device, and the target modules corresponding to the module identifiers are at least some modules corresponding to the events that need to be reproduced within the reproduction period.

[0014] Optionally, according to the reproduction request, obtaining the state information of each module in the device collected during the reproduction period and the event data corresponding to each module specifically includes:

[0015] According to the reproduction request, the state information of each target module in the device collected within the reproduction period and the event data corresponding to each target module are obtained.

[0016] Optionally, the event data includes at least one of communication timing, communication data and scheduling timing.

[0017] Optionally, according to the event data corresponding to each module, in the initial simulation environment, reproducing the events executed by each module during the operation of the device, specifically includes:

[0018] For each event, determining event data for the event;

[0019] When the event is a communication event, the sender module and the receiver module of the communication event are determined according to the communication timing contained in the event data, the initial simulation environment is configured as the triggering moment corresponding to the communication event, and the sender module is controlled to send the communication data contained in the event data to the receiver module;

[0020] When the event is a scheduling event, the scheduling module corresponding to the scheduling event is determined according to the scheduling timing contained in the event data, the initial simulation environment is configured as the triggering moment corresponding to the scheduling event, and the scheduling module is controlled to start running.

[0021] Optionally, controlling the sender module to send the communication data included in the event data to the receiver module specifically includes:

[0022] Determine whether the sender module belongs to the target module that needs to be reproduced;

[0023] If so, determining the data obtained by the operation of the sender module in the simulation environment as the communication data included in the event data, and controlling the sender module to send the communication data included in the event data to the receiver module;

[0024] If not, the sending module is controlled to send the communication data included in the event data to the receiving module.

[0025] Optionally, before reproducing the events executed by each module during the operation of the device, the method further includes:

[0026] For an event executed by each module at each moment, the triggering moment of the event is determined as the system time of the simulation environment.

[0027] Optionally, the device includes an unmanned device.

[0028] This specification provides a device for reproducing the operation process of a device, including:

[0029] A request receiving module, used for receiving a recurrence request, wherein the recurrence request at least includes a recurrence time period;

[0030] A data acquisition module, used to acquire, according to the reproduction request, the status information of each module in the device collected during the reproduction period, and the event data corresponding to each module;

[0031] An initial state determination module, used to determine the initial state of each module of the device according to the state information of each module, and to construct an initial simulation environment;

[0032] A triggering time determination module is used to determine the triggering time of each event according to the event data corresponding to each module;

[0033] The event reproducing module is used to reproduce the events executed by each module during the operation of the device in the initial simulation environment according to the triggering time of each event and the event data corresponding to each module.

[0034] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for reproducing the operation process of the device is implemented.

[0035] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for reproducing the operation process of the device when executing the program.

[0036] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0037] The method for reproducing the operation process of a device provided in this specification first receives a reproduction request that at least includes a reproduction time period, and then obtains the status information of each module in the device collected during the reproduction time period and the event data corresponding to each module according to the reproduction request, and determines the initial state of each module according to the status information of each module, builds an initial simulation environment, and determines the triggering time of each event according to the event data corresponding to each module. Finally, according to the triggering time of each event and the event data corresponding to each module, the events executed by each module during the operation of the device are reproduced in the initial simulation environment. Since the initial state of each module and the triggering time of each event that are consistent with the actual operation process are maintained during the simulation reproduction process, the reproduction process is highly consistent with the actual operation process, and the operation process of the device in the actual environment can be highly reproduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 A schematic diagram of a simulation reproduction process corresponding to an actual operation process provided in this specification;

[0040] Figure 2 A schematic diagram of a method flow chart for reproducing the operation process of a device provided in this manual;

[0041] Figure 3 A schematic diagram of a simulation system partial module reproducing event provided in this specification;

[0042] Figure 4 A simulation system data flow diagram provided in this manual;

[0043] Figure 5 A schematic diagram of a user-initiated reproduction request provided in this manual;

[0044] Figure 6 A schematic diagram of a device for reproducing the operation process of the equipment provided in this manual;

[0045] Figure 7 A schematic diagram of an electronic device for implementing a method for reproducing the operation process of a device provided in this specification. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this application will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] At present, in the simulation and reproduction process of the device, the events that occurred within the preset period of time when the problem occurred are executed in the simulation system in the order of the events, so as to simulate the operation process of the device in the actual environment when the problem occurs. Among them, the events mentioned include the scheduling between the modules of the device, the input and output of data, etc.

[0048] However, simply executing events in the order in which they occurred will result in an inaccurate reproduction of the actual operation process, and this difference will cause a large gap between the final reproduction result and the actual result in some edge cases. For example, whether the start time of the execution cycle of each simulation module is consistent, whether the initial operation state is consistent, whether the scheduling time of each module is completely consistent, etc., will affect the reproduction result. The gap between the reproduction result and the actual result may make it impossible to reproduce the operation process of the device in the actual environment where the problem occurs.

[0049] Figure 1 This is a schematic diagram of a simulation reproduction process corresponding to an actual operation process in this specification. Figure 1 The part above the dotted line represents the operation process of the perception algorithm module in the device in the actual environment, and the part below the dotted line represents the current simulation reproduction process corresponding to the above actual operation process. Among them, in the actual operation process diagram, the arrows with rectangular labels represent the inputs at different times, and the numbers on the rectangular labels represent the data. The numbers on the rectangular labels are used to refer to different input data in the following. The rectangular bar with time below the input data represents the scheduling event of the perception algorithm module, and the time on the rectangular bar represents the execution duration of the corresponding scheduling event. The horizontal axis with time scale under the scheduling event represents the timeline. The meaning of each element in the simulation reproduction process diagram below the dotted line is the same as that in the actual operation process diagram.

[0050] Depend on Figure 1It can be seen that the current simulation reproduction process is relatively idealized. The execution time of each scheduling event of the perception algorithm module is equal, and the execution of the scheduling event is not consistent with the input data, so that the data 3 that should have been input and processed in the first scheduling event is input and processed in the second scheduling event. The subsequent input of data 5, data 7, and data 8 are not consistent with the execution of the scheduling event. This will lead to a large difference in the final reproduction results in some edge cases, and it is impossible to reproduce the operation process of the device in the actual environment where the problem occurs. Therefore, the existing simulation reproduction is usually difficult to reproduce the problems that occur in edge cases.

[0051] The above situation is easy to occur in the scenario of reproducing the operation process of a device composed of multiple modules, for example, reproducing the operation process of a computer device, an unmanned driving device, etc. Among them, the above situation is particularly common when reproducing the operation process of an unmanned driving device. Therefore, for the convenience of description, this manual will be described later by taking the reproduction of the operation process of an unmanned driving device as an example.

[0052] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0053] Figure 2 This is a flow chart of a method for reproducing the operation process of a device in this specification, which specifically includes the following steps:

[0054] S100: receiving a recurrence request, wherein the recurrence request at least includes a recurrence period.

[0055] In actual applications, when the simulation system is performing simulation reproduction, a user may initiate a reproduction request, and the simulation system may receive the reproduction request and then perform reproduction according to the request data in the reproduction request.

[0056] Specifically, the simulation system can receive a reproduction request, and the reproduction request at least includes a reproduction time period. Among them, the reproduction request at least includes a reproduction time period, so that the simulation system can determine the start execution time and the end execution time of the subsequent simulation reproduction according to the reproduction time period, and obtain the data required in the time period for simulation. The reproduction time period can be determined as needed. For example, the reproduction time period can be a period of time before and after the device has a problem in actual operation (such as the first 50 seconds plus the last 10 seconds, a total of 1 minute). This specification does not limit how to determine the preset time period and the specific duration of the preset time period.

[0057] The simulation system mentioned in this specification refers to simulation software that can reproduce the operation process of the equipment. Taking unmanned driving equipment as an example, the simulation system includes Tencent Autonomous Driving Simulation (TAD Sim), RightHook, AirSim and other simulation software that can realize the simulation and reproduction of autonomous driving. Unmanned driving equipment includes unmanned passenger cars, drones, unmanned delivery equipment and other equipment that can realize autonomous driving. The execution subject of this specification is a terminal with simulation software installed, for example, a mobile phone, tablet computer, personal computer, etc. with simulation software installed. Of course, since the reproduction process is implemented in the simulation system, this specification does not limit the specific terminal used. For the convenience of explanation, the following description is only based on the simulation system as the execution subject.

[0058] S102: According to the reproduction request, the state information of each module in the device collected within the reproduction period and the event data corresponding to each module are obtained.

[0059] After receiving the recurrence request carrying the recurrence period, in one or more embodiments of the present specification, the simulation system may further obtain the state information of each module in the device collected during the recurrence period and the event data corresponding to each module according to the recurrence request, so that the simulation system can build a simulation environment based on the obtained state information and event data and further reproduce the device operation process.

[0060] The state information refers to the internal data required for the operation of each module in the device. Taking an unmanned driving device as an example, when the perception algorithm module analyzes obstacles near the unmanned driving device based on the images transmitted by the sensor, it may generate internal identifiers corresponding to each obstacle and object data corresponding to each identifier that records the obstacle information. The identifier and the object data that records the obstacle information are both the state information of the perception algorithm module.

[0061] Event data may include one or more of communication timing, communication data, and scheduling timing. The communication timing mentioned here may include the module corresponding to the sender, the module corresponding to the receiver, the sending time, the receiving time, and the processing time. Communication data may include data transmitted between modules in the device. Scheduling timing may include the start execution time and the end execution time when each module in the device executes an event.

[0062] It should be noted that the state information and event data obtained by the simulation system are all data with timestamps. The timestamp refers to the time corresponding to when the device stores the state, or the time corresponding to when communicating or scheduling. The simulation system can determine the state information and event data that need to be obtained based on the recurrence period and timestamp.

[0063] Furthermore, the simulation system can determine the start time of the recurrence period and the state of each module in the device according to the state data with timestamps during the simulation reproduction, and build the initial simulation environment. The simulation system can also determine the triggering time of each event according to the event data with timestamps during the simulation reproduction process, so as to reproduce and execute each event strictly according to each triggering time.

[0064] Specifically, when the device is running in an actual environment, the corresponding information such as communication and scheduling of each module during the operation process can be stored based on the time of occurrence, and the state corresponding to each time can also be stored based on the time. When performing simulation reproduction, the simulation system can obtain the state information and event data of each module in the reproduction period based on the reproduction period included in the reproduction request and the state information stored by each module during the actual operation of the device and the timestamp of the event data.

[0065] S104: Determine the initial state of each module of the device according to the state information of each module, and construct an initial simulation environment.

[0066] In one or more embodiments of the present specification, based on the state information of each module obtained above, the simulation system can determine the initial state of each module of the device and construct an initial simulation environment, so that the simulation system can perform subsequent simulation reproduction based on the same initial state as the device during actual operation.

[0067] Specifically, as can be seen from the description of step S102 above, the state information refers to the internal data required for the operation of each module in the device. Therefore, the simulation system can determine the initial state of each module of the device according to the internal data required for the operation of each module at the beginning of the recurrence period in the state information. The initial state can be understood as the state of each module after the internal data of the module is initialized according to the corresponding state information for each module. The initial simulation environment of the constructed device is the overall state of each module in the simulation system at the beginning of the recurrence period. When the simulation starts, the subsequent simulation environment is no longer the initial simulation environment.

[0068] For example, assuming the preset time period is 2021-01-01 9:00:00-9:05:00, the simulation system can initialize the internal data of the corresponding modules in the simulation system according to the internal data of each module at 2021-01-01 9:00:00 during the actual operation of the device in the status information, determine the initial state of each module of the device in the simulation system, and build a corresponding simulation environment.

[0069] S106: Determine the triggering time of each event according to the event data corresponding to each module.

[0070] After the initial simulation environment is built as described above, the simulation system can start to reproduce the operation process of the device according to the event data corresponding to each module.

[0071] Specifically, in one or more embodiments of the present specification, the simulation module may first determine the triggering time of each event according to the event data corresponding to each module.

[0072] Since in this specification, event data includes at least one of communication timing, communication data and scheduling timing, the simulation system can determine the triggering time of the communication event or scheduling event according to the communication timing and scheduling timing corresponding to each module.

[0073] For example, taking the communication timing shown in Table 1 as event data as an example, it can be determined that the triggering time of the communication event is 2021-01-01 9:00:05, and taking the scheduling timing shown in Table 2 as event data as an example, it can be determined that the triggering time of the scheduling event is 2021-01-01 9:00:06. Of course, there are actually a lot of event data corresponding to each module. For the convenience of description, only two examples are given here. In addition, it should be noted that for the convenience of description, the accuracy corresponding to each moment in Table 1 and Table 2 is only given to seconds. In actual applications, timestamps can be used to record each moment, and the accuracy can be determined as needed. Preferably, the accuracy corresponding to each moment can be nanoseconds. This specification does not limit how to record each moment and the accuracy corresponding to each moment.

[0074]

[0075] Table 1

[0076]

[0077] Table 2

[0078] S108: According to the triggering time of each event and the event data corresponding to each module, the events executed by each module during the operation of the device are reproduced in the initial simulation environment.

[0079] After determining the triggering time of each event, in one or more embodiments of the present specification, based on the triggering time of each event, the simulation system can reproduce the events executed by each module during the operation of the device in the initial simulation environment constructed in step S104 according to the event data corresponding to each module.

[0080] Among them, the initial simulation environment refers to the simulation environment at the beginning of the reproduction period, that is, the initial state of each module at the beginning of the reproduction process is maintained, which is consistent with the state of each module at that moment in the actual operation process. After the subsequent simulation starts, the simulation environment will also change accordingly, and it will no longer be the initial simulation environment.

[0081] Specifically, the simulation system can reproduce the events executed by each module during the operation of the device according to the triggering time of each event, so that the simulation system can reproduce the operation process of the device according to the sequence of each module executing each event during the actual operation of the device.

[0082] Furthermore, for each event to be reproduced, different reproduction processes may be performed according to the type of the event.

[0083] When the event is a communication event, the sender module, receiver module and communication data of the communication event are determined according to the communication timing and communication data corresponding to the communication event, and in the simulation environment, the sender module is controlled to send the communication data to the receiver module.

[0084] When the event is a scheduling event, the scheduling module corresponding to the scheduling event is determined according to the scheduling sequence corresponding to the event, and the scheduling module is controlled to start running in the simulation environment. In actual applications, the scheduling event is usually the execution of corresponding code.

[0085] It should be noted that for each event executed by each module at each moment, the simulation system can determine the triggering time of the event as the system time of the simulation environment. That is, by keeping the event execution time consistent with the actual operation process, the influence of the event execution time on the execution result is avoided, and the actual operation process of the device is highly reproduced.

[0086] Figure 3 This is a schematic diagram of a simulation system partial module reproducing events in this manual. Figure 3 In this example, unmanned driving equipment is used for illustration. The horizontal axis represents the time axis, which is from 2021-01-01 9:00:05 to 2021-01-01 9:00:12. Since the date is the same throughout the time axis, for the convenience of description, Figure 3 The date is omitted in the text. At the same time, due to the complexity of the overall operation process of unmanned driving equipment, Figure 3 Only two scheduling event examples for the perception algorithm module are given in the figure. Here, the first scheduling (i.e., the scheduling process from 9:00:05 to 9:00:08) is taken as an example for explanation.

[0087] First, before the scheduling event starts to execute, the simulation system sets the system time of the simulation environment to 2021-01-01 9:00:05.

[0088] Then, start to reproduce the running process corresponding to the scheduling event of the perception algorithm module. For example, the perception module may receive data transmitted from the sensor and store it in the cache device. Then, according to the communication timing of the corresponding communication event, the processing time of each data can be known. Then, the corresponding data can be input and processed according to the processing time.

[0089] by Figure 3 Taking data 1 in as an example, from the communication timing of the communication event corresponding to the data, it can be known that the processing time of the data 1 by the perception module is 2021-01-01 9:00:05, then at 2021-01-01 9:00:05 in this scheduling event, the data 1 is input and processed. After processing the data 1, from the communication timing of the communication event corresponding to the next data 2, it can be known that the processing time of the data 2 by the perception module is 2021-01-01 9:00:06. At this time, the simulation system can set the system time of the simulation environment to 2021-01-01 9:00:06, input data 2 and perform corresponding processing. And so on, until the scheduling event is completed, when the next scheduling event of the perception algorithm module is reproduced later, the simulation system can set the system time of the simulation environment to the start execution time of the next scheduling event.

[0090] Figure 3 For the convenience of description, only two scheduling event examples for the perception algorithm module are given. For communication events, for example, the sensor transmits the collected data to the perception algorithm module, Table 1 in step S106 is used as an example for explanation.

[0091] First, for the sensor, the corresponding communication event of the sensor should be executed within a certain scheduling event. Before the communication event starts to execute, the simulation system can first set the system time of the simulation environment to 2021-01-019:00:05.

[0092] Then, start to reproduce the operation process corresponding to the sensor in the communication event, that is, start to transmit the collected data to the perception algorithm module. When the transmission is completed, continue to execute the scheduling event corresponding to the communication event until the corresponding scheduling event is completed.

[0093] For the perception algorithm module of the receiver, the perception algorithm module only passively receives the data transmitted by the sensor through the cache device. When the cache device receives the data, the simulation system can set the system time of the simulation environment to 2021-01-01 9:00:06.

[0094] Then, the operation process corresponding to the cache device in the communication event begins to be reproduced, that is, the data transmitted by the sensor begins to be received until the transmission is completed.

[0095] Although only two scheduling events of the perception algorithm module and the communication events between the sensor and the perception algorithm module are used as examples here, in actual applications, the scheduling events and communication events corresponding to other modules of the unmanned driving device can also be applied to this description process. Then the simulation system can reproduce the events executed by each module during the operation of the unmanned driving device according to the event data corresponding to each module. In order to reproduce the operation process of the unmanned driving device. In addition, for the convenience of description, Figure 3 The accuracy of each moment in the simulation is only given to seconds. In practical applications, since the execution of events may be quite fast, the time accuracy can be determined as needed. Preferably, it can be nanoseconds. Then the simulation system can reproduce the events executed by each module in each nanosecond according to the triggering time of each event, so as to highly reproduce the entire operation process of the unmanned driving device.

[0096] based on Figure 2 The method for reproducing the operation process of the device shown first receives a reproduction request containing at least a reproduction period, and then obtains the state information of each module in the device collected within the reproduction period and the event data corresponding to each module according to the reproduction request. Then, according to the state information of each module, the initial state of each module is determined, an initial simulation environment is constructed, and the triggering time of each event is determined according to the event data corresponding to each module. Finally, according to the triggering time of each event and the event data corresponding to each module, the events executed by each module during the operation of the device are reproduced in the initial simulation environment. Since the initial state of each module and the triggering time of each event are kept consistent with the actual operation process during the simulation reproduction process, the reproduction process is highly consistent with the actual operation process, and the operation process of the device in the actual environment can be highly reproduced.

[0097] The simulation system described above may be a simulation software including a data reader, a state restorer and a scheduler. The simulation system can use the communication timing, the scheduling timing, the state data (i.e., the state information) and the communication data as simulation data to reproduce the operation process of the device. Figure 4 shown.

[0098] Figure 4 This is a schematic diagram of a simulation system data flow in this specification. Figure 4 In the embodiment, simulation data including communication timing, scheduling timing, state data and communication data, as well as a simulation system including a scheduler, a state restorer and a data reader are shown. The simulation system can be used to reproduce the operation process of a device composed of several modules. For example, when the device is an unmanned driving device, the simulation system can be used to reproduce the operation process of the unmanned driving device. This specification does not limit what kind of device needs to reproduce the operation process.

[0099] The data reader may read the communication data in the storage device in response to the scheduling of the scheduler.

[0100] The state restorer can respond to the scheduling of the scheduler and obtain the state data of each module in the device, so as to determine the initial state of each module of the device according to the state data of each module and construct an initial simulation environment.

[0101] The scheduler can schedule the state restorer according to the reproduction request, build an initial simulation environment, obtain the communication timing and the scheduling timing within the reproduction period, determine the triggering time of each event according to the communication timing and the scheduling timing, and reproduce the events executed by each module during the operation of the device in the initial simulation environment according to the triggering time of each event. Among them, the scheduling event is usually to execute the corresponding code, and for the communication event, the scheduler can schedule the data reader to obtain the communication data and transmit it to the required module.

[0102] Therefore, in one or more embodiments of this specification, based on Figure 2 The execution subject of the method for reproducing the operation process of the device shown can be a scheduler. Then step S100 can be a scheduler receiving a reproduction request, and the reproduction request at least includes a reproduction period. Step S102 can be a scheduler obtaining the communication timing and scheduling timing of each module in the device collected within the reproduction period according to the reproduction request. Step S104 can be a scheduler scheduling a state restorer to determine the initial state of each module of the device according to the state data of each module and build an initial simulation environment. Step S106 can be a scheduler determining the triggering time of each event according to the communication timing and scheduling timing corresponding to each module. Step S108 can be a scheduler reproducing the events executed by each module during the operation of the device in the initial simulation environment according to the triggering time of each event. Among them, when the event is a communication event, the scheduler schedules the data reader to read the corresponding communication data and transmit it to the corresponding module. The specific contents of each step have been described in detail above, and will not be repeated here one by one.

[0103] In addition, in one or more embodiments of the present specification, in step S100, the recurrence request may also include at least some module identifiers, and each target module corresponding to each module identifier is at least some modules corresponding to the event that needs to be reproduced within the recurrence period. Then, this recurrence may only reproduce the running process corresponding to the target module in the recurrence request.

[0104] This specification does not limit the specific content contained in the reproduction request. The reproduction request can be determined as needed. For example, this specification only uses scheduling events and communication events as examples to illustrate the reproduction process, but in actual applications, it is often more complicated, and other events may occur. Corresponding to other events, the reproduction request may also include other content, and this specification will not give examples one by one. Correspondingly, in step S102, the simulation system can obtain the status information of each target module in the device collected during the reproduction period, as well as the event data corresponding to each target module according to the reproduction request. The specific process can refer to the corresponding description in step S102.

[0105] Figure 5 This is a schematic diagram of a user-initiated reproduction request in this manual. Figure 5 As shown in , users can determine the recurrence period and check the modules to be reproduced on the request initiation page. Figure 2 The method for reproducing the operation process of the device shown can be understood as an example in which the user checks all module options in the reproduction module selection.

[0106] by Figure 5 For example, if module 1, module 3, and module 5 are checked in step S100, the reproduction request in step S100 may also include the module identifiers corresponding to these three modules, indicating that the user only reproduces the operation process of these three modules during this simulation reproduction process. Correspondingly, in step S102, the simulation system can obtain the status information of the three target modules in the device collected during the reproduction period, as well as the event data corresponding to the three target modules, according to the reproduction request.

[0107] Of course, in one or more embodiments of this specification, there are complex associations between the modules in the device. Therefore, the events that the user wants to reproduce within the recurrence period may also require the participation of other non-target modules. For example, during the operation of the perception algorithm module, the sensor may be required to provide the data it collects (such as images, three-dimensional point clouds) as support. However, the sensor is not the target module that the user wants to reproduce. Therefore, in step S102, the simulation system can also determine the relevant modules for executing the event according to the events that need to be reproduced within the recurrence period. Therefore, in this specification, the relevant modules include the target module and may also include non-target modules other than the target module. And obtain the status information of each relevant module in the device collected within the recurrence period, as well as the event data corresponding to each relevant module.

[0108] Further, in one or more embodiments of the present specification, corresponding to the above-mentioned reproducing the operation process of the target module only, in step S108, the simulation system can determine the events executed by each target module during the operation of the device, determine the related modules of the events executed by each target module, and reproduce the events of each target module based on the event data of each related module in the acquired event data.

[0109] For each event of each target module that needs to be reproduced, when the event is a communication event, the sender module and the receiver module of the communication event are determined according to the communication timing contained in the event data, the initial simulation environment is configured to the trigger moment corresponding to the communication event, and the sender module is controlled to send the communication data contained in the event data corresponding to the communication event to the receiver module. The communication event of the target module at that moment is reproduced. When the event is a scheduling event, the scheduling module corresponding to the scheduling event is determined according to the scheduling timing contained in the event data, the initial simulation environment is configured to the trigger moment corresponding to the scheduling event, the scheduling module is controlled to start running, and the scheduling event executed by the target module at that moment is reproduced.

[0110] Furthermore, in one or more embodiments of the present specification, when the event communication event to be executed at a certain moment is determined as described above, the simulation system can determine whether the sender module of the communication event belongs to the target module that needs to be reproduced. If so, determine the data obtained by running the sender module in the simulation environment as the communication data contained in the event data, or replace the communication data contained in the event data with the data obtained by running, and control the sender module to send the communication data contained in the event data to the receiver module. If not, control the sender module to send the communication data contained in the event data to the receiver module.

[0111] For example, take the operation process of the perception algorithm module receiving data transmitted by the sensor and determining the obstacles around the unmanned driving equipment as an example, where it is assumed that the perception algorithm module is the target module and the sensor is the non-target module. According to the determined triggering time of each event, for each event, determine the event data corresponding to the event. When it is determined that the first event in the execution process is a scheduling event of the perception algorithm, the simulation system can first determine the triggering time of the scheduling event as the system time of the initial simulation environment. Then execute the corresponding code corresponding to the scheduling event. For the specific reproduction process, please refer to the corresponding description in step S108. Since the simulation environment has undergone corresponding changes after executing the scheduling event, it is no longer the initial simulation environment subsequently.

[0112] Continuing with the assumption, when it is determined that the second event in the execution process is a communication event, the sender module of the communication event is determined to be the target module according to the sender in the communication sequence corresponding to the communication event. Obviously, the sensor is not the target module. At this time, the simulation system can first determine the receiving time in the communication sequence corresponding to the communication event as the system time of the simulation environment. Then determine the communication data contained in the event data corresponding to the communication event, and enable the cache device of the perception algorithm module to receive the communication data.

[0113] Of course, the sender module of the communication event may also be the target module. Then the simulation system can first determine the receiving time in the communication sequence corresponding to the communication event as the system time of the simulation environment. Then determine the data obtained by the target module running in the simulation system as or replace the communication data contained in the event data corresponding to the communication event, and control the target module to send the communication data contained in the event data to the cache device of the perception algorithm module.

[0114] In addition, the above contents are all explained by taking unmanned driving equipment as an example, but as long as the operation process of a device composed of several modules is reproduced, the method for reproducing the operation process of the device provided in this specification can be applied. For example, the device can also be a computer device. In this case, the above scheme can be executed by a terminal equipped with a simulation system corresponding to the computer device to reproduce the operation process of the computer device. This specification does not limit what kind of device needs to reproduce the operation process.

[0115] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0116] The method for reproducing the operation process of a device provided in this specification can be applied to the simulation reproduction of a device, such as a scenario in which a terminal with simulation software installed uses a simulation system to reproduce the operation process of a device. When the simulation system reproduces the operation process of a device in an actual environment, the method for reproducing the operation process of a device provided in this specification can be used to determine the simulation events that need to be executed by each simulation module of the simulation system at each moment, so as to highly reproduce the operation process of the device.

[0117] The above is a method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding device for reproducing the operation process of the device, such as Figure 6 shown.

[0118] Figure 6 A schematic diagram of a device for reproducing the operation process of the device provided in this manual includes:

[0119] A request receiving module 400 is used to receive a recurrence request, wherein the recurrence request at least includes a recurrence period;

[0120] The data acquisition module 402 is used to acquire the state information of each module in the device collected during the recurrence period and the event data corresponding to each module according to the recurrence request;

[0121] An initial state determination module 404 is used to determine the initial state of each module of the device according to the state information of each module and to construct an initial simulation environment;

[0122] A triggering time determination module 406, used to determine the triggering time of each event according to the event data corresponding to each module;

[0123] The event reproducing module 408 is used to reproduce the events executed by each module during the operation of the device in the initial simulation environment according to the triggering time of each event and the event data corresponding to each module.

[0124] Optionally, in the request receiving module 400, the reproduction request further includes module identifiers of at least some modules in the device, and the target modules corresponding to the module identifiers are at least some modules corresponding to the events that need to be reproduced within the reproduction period.

[0125] Optionally, the data acquisition module 402 acquires, according to the reproduction request, the state information of each target module in the device collected within the reproduction period and the event data corresponding to each target module.

[0126] Optionally, in the data acquisition module 402, the event data includes at least one of communication timing, communication data and scheduling timing.

[0127] Optionally, the recurring event module 408 determines the event data of each event. When the event is a communication event, the sender module and the receiver module of the communication event are determined according to the communication timing contained in the event data, the initial simulation environment is configured to the trigger moment corresponding to the communication event, and the sender module is controlled to send the communication data contained in the event data to the receiver module. When the event is a scheduling event, the scheduling module corresponding to the scheduling event is determined according to the scheduling timing contained in the event data, the initial simulation environment is configured to the trigger moment corresponding to the scheduling event, and the scheduling module is controlled to start running.

[0128] Optionally, the reproduction event module 408 determines whether the sender module belongs to the target module that needs to be reproduced. If so, it determines the data obtained by running the sender module in the simulation environment as the communication data contained in the event data, and controls the sender module to send the communication data contained in the event data to the receiver module. If not, it controls the sender module to send the communication data contained in the event data to the receiver module.

[0129] Optionally, the device also includes: a system time determination module 410, which is used to determine the triggering time of the event executed by each module at each moment before reproducing the events executed by each module during the operation of the device as the system time of the simulation environment.

[0130] The device for reproducing the operation process of a device provided in this specification can be used to reproduce the operation process of a device composed of several modules. For example, when the device is an unmanned driving device, the device can be used to reproduce the operation process of the unmanned driving device. This specification does not limit what kind of device needs to reproduce the operation process.

[0131] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 2 The computer program in the storage medium executes the method for reproducing the operation process of the device. Figure 2 The method for reproducing the operation process of a device provided can be used to reproduce the operation process of a device composed of several modules. For example, when the device is an unmanned driving device, the computer program can be used to reproduce the operation process of the unmanned driving device. This specification does not limit what kind of device needs to reproduce the operation process.

[0132] This manual also provides Figure 7 The structural diagram of the electronic device shown in FIG. Figure 7 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 2 The electronic device implements the method of reproducing the operation process of the device. Figure 2 The method for reproducing the operation process of a device can be used to reproduce the operation process of a device composed of several modules. For example, when the device is an unmanned driving device, the electronic device can be used to reproduce the operation process of the unmanned driving device. This specification does not limit what kind of device needs to reproduce the operation process.

[0133] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as the combination of logic device XOR software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic device.

[0134] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0135] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0136] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0137] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0138] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0142] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0143] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0144] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0145] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0146] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0148] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0149] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for reproducing the operation process of a device, characterized in that: include: receiving a recurrence request, wherein the recurrence request at least includes a recurrence time period; According to the reproduction request, obtaining the status information of each module in the device collected during the reproduction period and the event data corresponding to each module, wherein the event data includes at least one of a communication sequence, communication data and a scheduling sequence; According to the status information of each module, determine the initial status of each module of the device and build an initial simulation environment; Determine the triggering time of each event according to the event data corresponding to each module; According to the triggering time of each event and the event data corresponding to each module, in the initial simulation environment, the events executed by each module during the operation of the device are reproduced; The reproducing, in the initial simulation environment, the events executed by each module during the operation of the device according to the event data corresponding to each module specifically includes: For each event, determining event data for the event; When the event is a communication event, determining the sender module and the receiver module of the communication event according to the communication timing contained in the event data, configuring the initial simulation environment to the triggering moment corresponding to the communication event, and controlling the sender module to send the communication data contained in the event data to the receiver module; When the event is a scheduling event, determining a scheduling module corresponding to the scheduling event according to the scheduling timing contained in the event data, configuring the initial simulation environment to the triggering moment corresponding to the scheduling event, and controlling the scheduling module to start running; The controlling the sender module to send the communication data included in the event data to the receiver module specifically includes: determining whether the sender module belongs to a target module that needs to be reproduced; If so, determine the data obtained by the operation of the sender module in the simulation environment as the communication data contained in the event data, and control the sender module to send the communication data contained in the event data to the receiver module; if not, control the sender module to send the communication data contained in the event data to the receiver module.

2. The method according to claim 1, characterized in that The reproduction request further includes module identifiers of at least some modules in the device, and the target modules corresponding to the module identifiers are at least some modules corresponding to the events that need to be reproduced within the reproduction period.

3. The method according to claim 2, characterized in that According to the recurrence request, the state information of each module in the device collected during the recurrence period and the event data corresponding to each module are obtained, specifically including: According to the reproduction request, the state information of each target module in the device collected within the reproduction period and the event data corresponding to each target module are obtained.

4. The method according to claim 1, characterized in that Before reproducing the events executed by each module during the operation of the device, the method further includes: For an event executed by each module at each moment, the triggering moment of the event is determined as the system time of the simulation environment.

5. The method according to claim 1, characterized in that The equipment includes unmanned equipment.

6. A device for reproducing the operation process of a device, characterized in that: include: A request receiving module, used for receiving a recurrence request, wherein the recurrence request at least includes a recurrence time period; A data acquisition module, used to acquire, according to the reproduction request, the status information of each module in the device collected during the reproduction period, and the event data corresponding to each module, wherein the event data includes at least one of a communication sequence, communication data, and a scheduling sequence; An initial state determination module, used to determine the initial state of each module of the device according to the state information of each module, and to construct an initial simulation environment; A triggering time determination module is used to determine the triggering time of each event according to the event data corresponding to each module; An event reproducing module is used to reproduce the events executed by each module during the operation of the device in the initial simulation environment according to the triggering time of each event and the event data corresponding to each module; The reproducing, in the initial simulation environment, the events executed by each module during the operation of the device according to the event data corresponding to each module specifically includes: For each event, determining event data for the event; When the event is a communication event, determining the sender module and the receiver module of the communication event according to the communication timing contained in the event data, configuring the initial simulation environment to the triggering moment corresponding to the communication event, and controlling the sender module to send the communication data contained in the event data to the receiver module; When the event is a scheduling event, determining a scheduling module corresponding to the scheduling event according to the scheduling timing contained in the event data, configuring the initial simulation environment to the triggering moment corresponding to the scheduling event, and controlling the scheduling module to start running; The controlling the sender module to send the communication data included in the event data to the receiver module specifically includes: determining whether the sender module belongs to a target module that needs to be reproduced; If so, determine the data obtained by the operation of the sender module in the simulation environment as the communication data contained in the event data, and control the sender module to send the communication data contained in the event data to the receiver module; if not, control the sender module to send the communication data contained in the event data to the receiver module.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images