Networked path planning and scheduling method and device for traversal learning based on video stream

Abstract

The present invention discloses a method and device for networked path planning and scheduling of crossing learning based on video stream, and the method includes: S1: establishing a path planning and scheduling network; S2: obtaining relevant monitoring video streams; S3: building a map based on the video stream; S4: correlating the dynamics of the environment and the dynamics of the mobile terminal in the video stream to perform relative positioning navigation; S5: real-time collaborative path planning and scheduling based on the network of multiple mobile terminals of the video stream. The device includes: at least one processor, a server and a memory; the memory stores computer execution instructions; the server establishes a network core node for path planning and scheduling; the processor executes the computer execution instructions stored in the memory, so that the processor executes the networked path planning and scheduling method for crossing learning based on video stream. The advantages of the present invention are: realizing real-time path planning and scheduling in a complex and dynamic environment in space to help coordinate, coordinate and guide various safe movements or flights.

Description

Technical Field

[0001] The present invention relates to the technical field of networked path planning and scheduling, and in particular to a networked path planning and scheduling method and device for crossing learning based on video stream. Background Art

[0002] Real-time path planning and scheduling are the key to ensure the stable and safe operation of highly dynamic mobile network systems. They are crucial and are the frontier, hot spot and difficulty of research in the field of air, space, land and sea information networks. With the vigorous development of air, space, land and sea information networks, various intelligent mobile terminals are rapidly changing people's lives, such as planets, satellites, aircraft, ships, cars, mobile phones, hot air balloons, various military and civilian robots, etc., which are developing in full swing. "Internet + Robots" is gradually upgrading and the demand is very strong. At the same time, the complex and changeable environment space of many application scenarios, high dynamics, and many uncertain factors have posed new challenges to the real-time path planning and scheduling of mobile terminals, which has attracted widespread attention from the international academic and industrial circles. The main difficulties faced are large and complex data, fast changes in time and space, high requirements for real-time information processing, and high requirements for safety and reliability. About 80% of human information is obtained by vision, that is, in our new era of intelligence, video streaming will play a major role in liberating humans to obtain intelligent information. In addition to video streams obtained by video sensor devices such as cameras, sensor signal information such as lidar can also be quickly converted into video streams through calculations. Video streams can be converted into data in the form of text, sound, images, symbols, etc., and have the characteristics of flexibility, uniformity, and transmittability. Path planning and scheduling based on video streams will be the main trend of the development of intelligent path planning and scheduling technology in the future. Real-time path planning and scheduling technology is relatively lacking, and research on it is still in its infancy.

[0003] In the prior art, path planning and scheduling are generally used more in simple environments, and have achieved certain results in dynamic environments, but they require large computing resources, have poor real-time performance, low robustness, and poor stability. At present, most of the literature on joint research with navigation in path planning and scheduling methods uses static environment navigation data or dynamic known environment navigation data for path planning and scheduling, such as GPS navigation data, Beidou navigation data, etc. In recent years, navigation in dynamic unknown environments has also achieved certain results, such as visual navigation, but in the face of environmental complexity, it is still necessary to continue in-depth exploration and research, and there is still a lack of joint research on path planning and scheduling and navigation in dynamic unknown environments, which will cause greater safety hazards when mobile terminals move or fly. Especially in dynamic environment path scheduling, joint research with navigation is very necessary.

[0004] In terms of multi-mobile terminal collaborative path planning and scheduling, existing technologies have many problems, such as high prior knowledge requirements, single consideration factors, high latency, simple environment structure, many path conflicts, low accuracy, and poor system coordination. There are relatively few studies on networked systems for multi-mobile terminal collaborative path planning and scheduling, and collaborative strategies are also scarce. The security and reliability are not high, which deserves and needs in-depth research. Summary of the invention

[0005] In view of the defects of the prior art, the present invention provides a video stream-based traversal learning networked path planning and scheduling method and device.

[0006] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0007] A video stream-based traversal learning networked path planning and scheduling method comprises the following steps:

[0008] S1: Establish path planning and scheduling network;

[0009] S2: Obtain relevant monitoring video stream;

[0010] S3: Build a map based on the video stream;

[0011] S4: Relative positioning and navigation based on the dynamics of the environment and the mobile terminal in the associated video stream;

[0012] S5: Real-time collaborative path planning and scheduling based on video streams for multiple mobile terminals in a network.

[0013] Furthermore, the S1 establishes a path planning and scheduling network, including the following steps:

[0014] Build network software and hardware infrastructure, including: path planning and scheduling network servers, content, storage, routers and gateways;

[0015] Establish a route planning and scheduling video stream intelligent processing network: build it separately on the cloud or mobile terminal, or build it together with the cloud and mobile terminals to form a networked system;

[0016] The networked system is divided into storage layer, test layer, training layer and application layer;

[0017] The training layer is included in the test layer to ensure the consistent and correct availability of data;

[0018] The storage layer is used to store video streams and theories, models, algorithms, and data that have passed training and testing;

[0019] The test layer tests and processes the video stream according to the corresponding services;

[0020] The training layer performs training processing on the video streams that pass the test according to the corresponding services;

[0021] The application layer issues service requests and receives service information based on the corresponding services of mobile terminal path planning and scheduling. The service information includes: environment modeling, map construction, route planning, route adjustment, dynamic prediction and specific requirements.

[0022] Further, the S2 acquires relevant monitoring video streams, including: acquiring environmental monitoring video streams, network dynamic topology video streams, and network transmission related video streams; acquiring video streams includes acquiring historical video streams, current video streams, and digital video streams;

[0023] The acquisition methods include: acquiring from the network, acquiring from the sensor of the mobile terminal camera, acquiring from the sensor of a certain area, acquiring from the relevant sensor of a certain device, and acquiring from the sensor of a certain equipment;

[0024] The video stream includes: frame images, voice, and text language description related information.

[0025] Furthermore, S3 builds a map based on the video stream including:

[0026] Analyze the internal connections between the physical environment (real environment), related monitoring video streams, path categories and mobile terminals, and build a dynamic map in real time for mobile terminals to move or fly safely to the target space;

[0027] The dynamic map can be updated in real time according to the changes in the video stream path environment;

[0028] Dynamic maps include: feasible paths, routes, boundaries, related things and / or objects and descriptions, which include: dimensions, text and symbols;

[0029] According to the current video stream, the current map is constructed in real time by comparing and matching with the existing digital map of the current area's physical environment and / or digital video stream, historical video stream, and related information. The related information includes: images, voice, text language, and data through identification and reference.

[0030] Furthermore, the S4 associates the dynamics of the environment in the video stream with the dynamics of the mobile terminal to perform relative positioning navigation, including:

[0031] Analyze the differences in video streams at different time windows, analyze the spatiotemporality of video streams from a global perspective, establish associations between environmental obstacles and / or environmental objects (environmental objects include static objects and dynamic objects in the environment) and the actual physical world, and obtain their spatiotemporality and actual physical position and size, density, and weight characteristics in the actual physical world; analyze the differences in multi-associated frame images of video streams, and obtain the relative dynamic changes of local environmental obstacles and / or environmental objects based on the differences and using static and / or dynamic objects or objects as relative references, and measure and predict the dynamic characteristics of obstacles and / or related objects in combination with the actual physical position and characteristics of reference objects or environmental obstacles and / or environmental objects. The dynamic characteristics include: speed, acceleration, moving route, posture, and structural changes. Structural changes include structural changes between various related objects in a certain area, structural changes of a certain object, and structural changes of a certain type of object;

[0032] Analyze the differences of multiple associated frame images of the video stream, and perform real-time dynamic measurement and prediction of characteristic data of obstacles and / or roads, routes, movable path boundaries, key distances, and key navigation points based on the differences and taking static and / or dynamic things or objects as relative references, and combining the actual physical position and size, density, and weight characteristics of the references;

[0033] For video streams obtained by sensors such as cameras of the same type, when the sensor parameter settings remain unchanged, the reference object is the relevant object in the historical video stream or the current video stream. When the sensor parameter settings change, the reference object and the relevant object in the current video stream are transformed to the same standard for reference through parameter calculation and conversion.

[0034] For video streams obtained by sensors such as different types of cameras, the reference object and the related object in the current video stream are transformed to the same standard for reference by calculating and converting different types of parameters. The reference object is the related object in the obtained historical video stream or the current video stream.

[0035] Based on the measurement and prediction of the dynamic characteristics of environmental obstacles and / or environmental objects, the relative positioning information of the environmental dynamics and the mobile terminal dynamics in the video stream is obtained, and the path and navigation route are planned based on the relative positioning information and the road size and destination feature information, so that the mobile terminal can move or fly safely to the target space.

[0036] Furthermore, the S4 associates the dynamics of the environment in the video stream with the dynamics of the mobile terminal to perform relative positioning navigation, and further includes:

[0037] Analyze the spatiotemporal, correlation, difference, and content mapping issues of video streams, classify or cluster video streams, and establish a mapping cloud database based on and statistically calculate the spatiotemporal mapping methods, mapping content, mapping efficiency, and mapping accuracy of various types of video stream environments and physical environments;

[0038] At the same time, the current physical environment, monitoring video stream, path category, mobile terminal and mapping cloud database are associated to form a path planning and scheduling video stream understanding and cognition system;

[0039] Based on the understanding and cognitive system, the dynamic characteristics of the current environment obstacles and / or objects are calculated, measured and predicted, the current obstacles and / or objects and the mobile terminal are dynamically relatively positioned, and based on the relative positioning information and target space information, the navigation method and navigation route that match the current mobile terminal motion characteristics are calculated and predicted;

[0040] According to the spatiotemporal phase mapping between the video stream environment and the physical environment and the video stream intelligent processing network, the internal connections between the four elements of physical environment, monitoring video stream, path category and mobile terminal are analyzed, and adaptive continuous smooth variable angle dynamic environment modeling and algorithm optimization for path planning and scheduling are carried out, and a map rapid construction and update algorithm for the mobile terminal to safely move or fly to the target space is established; and or a 360-degree continuous smooth variable angle environment modeling algorithm is established for modeling.

[0041] Furthermore, the S5 is based on real-time collaborative path planning and scheduling of multiple mobile terminals in a video stream network, including:

[0042] Establish a web-based real-time collaborative route planning and scheduling app and / or related software, including applets and navigation device embedded software;

[0043] Design the path planning and dispatching network system as a blockchain-based named data network security architecture, combining the naming scheme with the blockchain;

[0044] Combine naming schemes, semantic information and blockchain to securely transmit video streams in real time;

[0045] Combine naming schemes, semantic information and privacy protection technology to improve the security and confidentiality of information transmission;

[0046] Semantic information includes: text, speech, symbols, graphics and images;

[0047] Analyze and establish a blockchain named data network security architecture for path planning and scheduling based on the logical relationship and structural layout between the path planning and scheduling video stream intelligent processing network and the blockchain-based named data network security architecture and the relevant areas or related spaces where the physical location of the road is located;

[0048] According to the actual movement or flight status of multiple mobile terminals, environmental monitoring video streams, mid-way paths, mid-way target spaces, and destinations, the dynamic changes of the network topology are predicted. At the same time, based on node correlation and information routing transmission, multiple mobile terminals are networked and positioned in real time to obtain the location of the mobile terminals.

[0049] The positioning information is combined with the environmental monitoring video stream and the actual physical environment associated with the location of the mobile terminal, and a map is constructed based on the video stream. The dynamics of the environment in the video stream and the dynamics of the mobile terminal are associated to perform positioning navigation and continue path planning and scheduling;

[0050] The dynamic changes of the network topology structure, network node correlation, information routing transmission, real-time positioning of multiple mobile terminals and related data of the networked system for cooperative path planning and scheduling of multiple mobile terminals are video streamed, and the video stream of the video stream is correlated and matched with the corresponding spatial complex dynamic environment monitoring video stream;

[0051] Analyze video streams, and study algorithms and collaborative methods for collaborative real-time path planning and scheduling of multiple mobile terminals based on their different characteristics, functions, states, environments, and destinations, such as bird flock network topology, ant colony network topology, distributed network topology, centralized network topology, chain network topology, and traffic network network topology;

[0052] According to the real-time path planning and scheduling demand categories, collaborative network systems and network environments, video stream intelligent processing network training and testing conditions, etc., collaborative optimization is performed on the real-time path planning and scheduling of multi-mobile terminal networks based on video streams.

[0053] Furthermore, the path planning and scheduling network is established by adopting the traversal learning technology;

[0054] Use traversal learning technology to establish a path planning and scheduling video stream intelligent processing network;

[0055] Use cross-learning technology to build maps in real time;

[0056] Use traversal learning technology to perform relative positioning and navigation based on the dynamics of the environment and the mobile terminal;

[0057] The traversal learning technology is used to calculate and predict the relative positioning information and actual physical positioning information of relevant objects and mobile terminals in the video stream;

[0058] Using traversal learning technology to calculate and predict the movement routes, navigation methods and navigation routes of relevant objects and mobile terminals in the video stream;

[0059] Cross-learning technology is used to perform real-time collaborative path planning and scheduling for multiple mobile terminals in a network.

[0060] Furthermore, the cross-learning refers to learning the same or different scenes and events and their global or local features and related knowledge across time and / or space, and solving problems through intelligent reference, matching, calculation and prediction. The time can be a single or multiple different continuous time periods, the space can be a single or multiple different places or continuous spaces, and the event can be a single or multiple events, wherein the important manifestations of the event include video streams, voice and text language descriptions, and the event includes continuously changing events, discretely changing events and continuously discrete mixed changing events;

[0061] According to application requirements, etc., the through-learning can be combined with intelligent computing technologies such as mapping learning, deep learning, reinforcement learning, and statistical learning to form a new through-learning technology. The through-learning technology belongs to multi-source learning technology.

[0062] Furthermore, the video stream intelligent processing network is an organic combination of a wide neural network and a deep neural network for intelligent processing of video streams;

[0063] The theories, models, algorithms, and methods at each layer of the intelligent processing network constitute the components of the breadth neural network and deep neural network for intelligent processing of video streams for path planning and scheduling. The various operating composition modes between the components are studied in parallel, serial, cyclic, random, probabilistic, and statistical ways, making the video stream intelligent processing network a computing system and data center that can perform cross-learning intelligent processing of video streams for path planning and scheduling.

[0064] The present invention also discloses a video stream-based traversal learning networked path planning and scheduling system, comprising: a configuration module, an acquisition module, an environment modeling module, a map generation module, a navigation module, a path planning and scheduling module and a visualization module;

[0065] The configuration module is used to establish a configuration path planning and scheduling network;

[0066] The acquisition module is used to acquire relevant monitoring video streams;

[0067] The environment modeling module is used to analyze the internal connection between the four elements of physical environment, monitoring video stream, path category and mobile terminal according to the spatiotemporal phase mapping between the video stream environment and the physical environment and the video stream intelligent processing network, and to perform adaptive continuous smooth variable angle dynamic environment modeling for path planning and scheduling;

[0068] The map generation module is used to construct a map in real time based on the spatiotemporal phase mapping of the video stream environment and the physical environment and the video stream intelligent processing network, as well as the data and / or results of the environment modeling module;

[0069] The navigation module is used to associate the dynamics of the environment and the dynamics of the mobile terminal in the video stream to perform relative positioning navigation;

[0070] The path planning and scheduling module is used for real-time collaborative path planning and scheduling based on video stream multi-mobile terminal networking;

[0071] The visualization module is used to display the data and / or results generated by the configuration module, acquisition module, environment modeling module, map generation module, navigation module, path planning and scheduling module for viewing, and send the display effect and demand real-time feedback notification to the above modules, so that the above modules can adjust and optimize the algorithms, parameters and models in real time, and then display the data and / or results; the data and / or results generated by the above modules can be displayed in split screens and pages, or integrated on the same screen for display;

[0072] The configuration module, acquisition module, environment modeling module, map generation module, navigation module, path planning and scheduling module, and visualization module are interconnected and can perform data sharing, data mutual transmission, and real-time mutual feedback of information.

[0073] Further, the configuration module is used to establish a configuration path planning and scheduling network using traversal learning technology;

[0074] The acquisition module is used to acquire current environment monitoring video stream, digital video stream, historical video stream data and information from mobile terminals, networks, and sensors at various locations;

[0075] The environment modeling module is used to perform real-time environment modeling using the traversal learning technology;

[0076] The map generation module is used to construct a dynamic map in real time using the traversal learning technology;

[0077] The navigation module is used to use the traversal learning technology to associate the dynamics of the environment and the dynamics of the mobile terminal in the video stream to perform relative positioning navigation, calculate and predict the relative positioning information and actual physical positioning information of the relevant objects and the mobile terminal in the video stream, and calculate and predict the movement route, navigation method and navigation route of the relevant objects and the mobile terminal in the video stream;

[0078] The path planning and scheduling module is used to utilize the traversal learning technology to perform real-time collaborative path planning and scheduling based on the video stream multi-mobile terminal networking.

[0079] The present invention also discloses a video stream-based traversal learning networked path planning and scheduling device, comprising: at least one processor, a server and a memory;

[0080] The memory stores computer-executable instructions;

[0081] The server establishes a network core node for path planning and scheduling;

[0082] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs a video stream-based traversal learning networked path planning and scheduling method.

[0083] Compared with the prior art, the advantages of the present invention are:

[0084] Break through the development bottleneck of real-time path planning and scheduling based on video streaming in complex and dynamic spatial environments, so that various mobile terminals, robots, etc. can achieve real-time path planning and scheduling in complex and dynamic spatial environments, to help coordinate, coordinate and guide various safe movements or flights. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A flow chart of a video stream-based traversal learning networked path planning and scheduling method provided in an embodiment of the present invention;

[0086] Figure 2 A schematic diagram of the process flow of a video stream understanding and recognition system according to an embodiment of the present invention;

[0087] Figure 3 A schematic diagram of a network structure for intelligent processing of video streams according to an embodiment of the present invention;

[0088] Figure 4 This is a schematic diagram of the basic structure of video stream processing for learning through an embodiment of the present invention;

[0089] Figure 5 A schematic diagram of the basic structure of the dynamic characteristics measurement and prediction process of the dynamic environment of the video stream through learning according to the embodiment of the present invention;

[0090] Figure 6 A basic structural diagram of real-time positioning navigation, path planning and scheduling of multiple mobile terminals in a network according to an embodiment of the present invention;

[0091] Figure 7 A schematic diagram of the basic structure of multi-mobile terminal collaboration real-time path planning and scheduling video stream processing according to an embodiment of the present invention;

[0092] Figure 8 A schematic diagram of the connection modules of a networked path planning and scheduling device for traversal learning based on video streams provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0094] like Figure 1As shown, a video stream-based traversal learning networked path planning and scheduling method includes the following steps:

[0095] Establishing a path planning and scheduling network;

[0096] Obtain relevant monitoring video streams;

[0097] Build maps based on video streams;

[0098] Relative positioning navigation is performed by associating the dynamics of the environment and the mobile terminal in the video stream;

[0099] Real-time collaborative path planning and scheduling for multiple mobile terminals in a network based on video streams.

[0100] The establishment of the path planning and scheduling network includes:

[0101] Path planning and scheduling network servers, content, storage, routers, gateways and other network software and hardware infrastructure construction;

[0102] Establishing intelligent processing network for path planning and scheduling video streams;

[0103] The video stream intelligent processing network is an organic combination of a wide neural network and a deep neural network for intelligent processing of video streams for path planning and scheduling;

[0104] The video stream intelligent processing network is a computing system and data center that can perform intelligent processing of video streams through learning for path planning and scheduling;

[0105] It can be built in the form of a cloud platform, or built separately for a specific mobile terminal, or built together by networking the cloud and mobile terminals to form a networked system for real-time path planning and scheduling of multiple mobile terminals based on video streams;

[0106] The path planning and scheduling video stream intelligent processing network is divided into storage layer, test layer, training layer and application layer;

[0107] Cross-learning refers to the theoretical technology and / or method of learning the same or different scenes and events and their global or local features and related knowledge across time or space, and solving problems through intelligent reference, matching, calculation and prediction. Time can be a single or multiple different continuous time periods, space can be a single or multiple different places or continuous spaces, and events can be a single or multiple events. Events include continuously changing events, discretely changing events, and continuous discrete mixed changing events. The important manifestations of events are video streams, voice and text language descriptions. According to application requirements, cross-learning can be combined with intelligent computing technologies such as mapping learning, deep learning, reinforcement learning, and statistical learning to form new cross-learning technologies. Cross-learning technologies belong to multi-source learning technologies.

[0108] The obtaining of the relevant monitoring video stream comprises:

[0109] Obtain environmental monitoring video streams, obtain network dynamic topology video streams, obtain network transmission related video streams, etc.;

[0110] Acquired from the network, from mobile terminal cameras and other related sensors, from a certain device or equipment in a certain area or section, and other related sensors;

[0111] Acquiring video streams includes acquiring relevant information such as historical video streams, real-time video streams, and digital video streams;

[0112] The video stream may include frame images, voice, text language description and other related information.

[0113] The map construction based on the video stream includes:

[0114] Analyze the internal connection between the four elements of physical environment, monitoring video stream, path category and mobile terminal, and build a dynamic map in real time for the mobile terminal to move or fly safely to the target space; path types include roads, railways, dirt roads, waterways, air routes, tunnels, pipelines, indoors, outdoors, forests, space, interstellar and other relevant roads and / or spaces where mobile terminals can move or fly;

[0115] The map can be updated in real time according to changes in the video stream path environment;

[0116] The map may include possible paths, routes, boundary lines, related things and / or objects and their related dimensions and descriptions such as related words, symbols, etc.;

[0117] According to the current video stream, based on the existing relevant digital map and / or digital video stream, relevant historical video stream, relevant information, etc., the current map can be constructed in real time through identification, reference, comparison and matching;

[0118] Use the idea of ​​​​cross-learning to build maps in real time.

[0119] The relative positioning navigation based on the dynamics of the environment and the dynamics of the mobile terminal in the associated video stream includes:

[0120] Analyze the differences in video streams at different time windows, analyze the spatiotemporality of video streams from a global perspective, establish associations between environmental obstacles and / or environmental objects such as roads and the actual physical world, and obtain their spatiotemporality and actual physical position and size, density, weight and other characteristics in the actual physical world; analyze the differences in multi-associated frame images of video streams, and obtain the relative dynamic changes of local environmental obstacles and / or environmental objects based on the differences and using static and / or dynamic things or objects as relative references, and measure and predict the dynamic characteristics of obstacles and / or related objects, such as speed, acceleration, and moving routes, in combination with the actual physical position and characteristics of reference objects or environmental obstacles and / or environmental objects; wherein environmental related objects include static objects and dynamic objects in the environment;

[0121] Analyze the differences of multiple associated frame images of the video stream, and perform real-time dynamic measurement and prediction of obstacles and related objects, as well as roads, routes, movable path boundaries, key distances, key navigation points and other feature data based on the differences and taking static and / or dynamic things or objects as relative references, and combining the actual physical position and size, density, weight and other characteristics of the reference objects;

[0122] For video streams obtained by the same type of camera sensor, when the sensor parameter settings remain unchanged, the reference object is the relevant object in the obtained historical video stream or the current video stream. When the sensor parameter settings change, the reference object and the relevant object in the current video stream are transformed to the same standard for reference through parameter calculation conversion;

[0123] For video streams obtained by different types of camera sensors, the reference object and the related object in the current video stream are transformed to the same standard for reference through different types of parameter calculation conversion, and the reference object is the related object in the obtained historical video stream or the current video stream;

[0124] Relevant information such as road dimensions can also be obtained based on prior knowledge;

[0125] According to the dynamic characteristics measurement and prediction of obstacles and / or related objects, the relative positioning information of the dynamic environment and the dynamic mobile terminal in the video stream is obtained, and the path and navigation route are planned according to the relative positioning information and characteristic information such as road size and destination, so that the mobile terminal can move or fly to the target space safely;

[0126] Analyze the spatiotemporal, correlation, difference, content mapping and other issues of large-scale video streams, classify or cluster video streams, and establish a mapping cloud database based on and statistically calculate the spatiotemporal mapping methods, mapping content, mapping efficiency and mapping accuracy of various types of video stream environments and physical environments;

[0127] At the same time, the current physical environment, monitoring video stream, path category and mobile terminal are associated with the cloud database to form a path planning and scheduling video stream understanding and cognition system;

[0128] According to this system, based on the idea of ​​"crossing learning", the dynamic characteristics of the current environmental obstacles and / or environmental objects are calculated, measured and predicted, the current obstacles and / or related objects and the mobile terminal are dynamically relatively positioned, and based on the relative positioning information and target space information, the navigation method and navigation route that match the current mobile terminal's motion characteristics are calculated and predicted.

[0129] According to the spatiotemporal phase mapping between the video stream environment and the physical environment and the video stream intelligent processing network, the internal connections between the four elements of the physical environment, monitoring video streams, path categories and mobile terminals are analyzed, and adaptive continuous smooth variable angle dynamic environment modeling and algorithm optimization for path planning and scheduling are carried out. An algorithm for quickly building and updating maps for mobile terminals to safely move or fly to the target space is established to improve the real-time forward accuracy and safety flexibility of mobile terminals; and or a 360-degree continuous smooth variable angle environment modeling algorithm is established for modeling.

[0130] The real-time collaborative path planning and scheduling based on video stream multi-mobile terminal networking includes:

[0131] Establish a web-based real-time collaborative route planning and scheduling app and / or related software;

[0132] The network system is designed as a blockchain-based named data network security architecture, combining naming with blockchain to improve the efficiency and security of multi-mobile terminal video streaming transmission;

[0133] Combine naming, semantics and blockchain to achieve real-time secure transmission of video streams;

[0134] Combine naming schemes, semantic information and privacy protection technology to improve the security and confidentiality of information transmission;

[0135] Semantic information includes information data in various forms such as text, speech, symbols, graphics, and images;

[0136] Analyze and establish a blockchain named data network security architecture for path planning and scheduling based on the logical relationship and structural layout between the path planning and scheduling video stream intelligent processing network and the blockchain-based named data network security architecture and the relevant areas or related spaces where the physical location of the road is located;

[0137] According to the actual movement or flight status of multiple mobile terminals, environmental monitoring video streams, mid-way paths, mid-way target spaces, destinations, etc., the dynamic changes of the network topology are predicted, and the networked real-time positioning of multiple mobile terminals is performed based on node correlation and information routing transmission; the movement or flight status includes the movement or flight trajectory, attitude, speed, acceleration, direction, fuel consumption, and future movement or flight trends, etc.;

[0138] The positioning information is combined with the environment monitoring video stream associated with the location of the mobile terminal and the actual physical environment, and a map is constructed according to the video stream. The environment dynamics in the associated video stream and the dynamics of the mobile terminal are used for positioning navigation to continue path planning and scheduling;

[0139] The dynamic changes of the network topology structure, network node correlation, information routing transmission, and real-time positioning of multiple mobile terminals in the networked system for collaborative path planning and scheduling of multiple mobile terminals are streamed, and the video stream is associated and matched with the corresponding spatial complex dynamic environment monitoring video stream to jointly form the basic environmental elements of "cross-learning" for the collaborative path planning and scheduling of multiple mobile terminals in the network;

[0140] Analyze video streams, and based on the idea of ​​"cross-learning", study algorithms and collaborative methods for collaborative real-time path planning and scheduling of multiple mobile terminals according to their different characteristics, functions, states, environments, destinations, etc., such as bird flock network topology method, ant colony network topology method, distributed network topology method, centralized network topology method, chain network topology method, traffic network network topology method, etc.;

[0141] According to the real-time path planning and scheduling demand categories, collaborative network systems and network environments, video stream intelligent processing network training and testing conditions, etc., collaborative optimization is performed on the real-time path planning and scheduling of multi-mobile terminal networks based on video streams.

[0142] A video stream-based traversal learning networked path planning and scheduling system, characterized by comprising:

[0143] A configuration module, the configuration module is used to establish a configuration path planning and scheduling network;

[0144] An acquisition module, the acquisition module is used to acquire relevant monitoring video streams;

[0145] An environment modeling module, which is used to analyze the internal connections between the four elements of the physical environment, monitoring video stream, path category and mobile terminal based on the spatiotemporal phase mapping between the video stream environment and the physical environment and the video stream intelligent processing network, and to perform adaptive continuous smooth variable angle dynamic environment modeling for path planning and scheduling;

[0146] A map generation module, the map generation module is used to construct a map in real time based on the spatiotemporal phase mapping of the video stream environment and the physical environment and the video stream intelligent processing network, as well as the data and / or results of the environment modeling module;

[0147] A navigation module, the navigation module is used to associate the dynamics of the environment in the video stream with the dynamics of the mobile terminal to perform relative positioning navigation;

[0148] A path planning and scheduling module, which is used for real-time collaborative path planning and scheduling based on video stream multi-mobile terminal networking;

[0149] A visualization module, the visualization module is used to display the data and / or results generated by the configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, and the path planning and scheduling module for viewing, and send real-time feedback notifications such as display effects and requirements to the above modules, so that the above modules can adjust and optimize algorithms, parameters, models, etc. in real time, and display the data and / or results again; the data and / or results generated by the above modules can be displayed in separate screens and pages, or can be integrated on the same screen for display;

[0150] The configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, the path planning and scheduling module, and the visualization module are interconnected and can perform data sharing, data mutual transmission, and real-time mutual feedback of information.

[0151] The configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, and the path planning and scheduling module are specifically used to:

[0152] A configuration module, wherein the configuration module is used to establish a configuration path planning and scheduling network using a traversal learning technique;

[0153] An acquisition module, which is used to acquire data and information such as current environment monitoring video stream, digital video stream, historical video stream, etc. from mobile terminals, networks, sensors at various locations, etc.;

[0154] An environment modeling module, wherein the environment modeling module is used to perform real-time environment modeling using a traversal learning technique;

[0155] A map generation module, wherein the map generation module is used to construct a dynamic map in real time using a traversal learning technique;

[0156] A navigation module, the navigation module is used to use the traversal learning technology to associate the dynamics of the environment and the dynamics of the mobile terminal in the video stream to perform relative positioning navigation, calculate and predict the relative positioning information and actual physical positioning information of the relevant objects and the mobile terminal in the video stream, calculate and predict the movement route, navigation method and navigation route of the relevant objects and the mobile terminal in the video stream;

[0157] A path planning and scheduling module is used to utilize traversal learning technology to perform real-time collaborative path planning and scheduling based on video stream multi-mobile terminal networking.

[0158] A video stream-based traversal learning networked path planning and scheduling device, comprising: a processor, a server and a memory;

[0159] The memory stores computer-executable instructions;

[0160] The server establishes a network core node for path planning and scheduling;

[0161] The at least one processor executes the computer-executable instructions stored in the memory, so that the processor executes the video stream-based traversal learning networked path planning and scheduling method.

[0162] A computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, a video stream-based traversal learning networked path planning and scheduling method is implemented.

[0163] like Figure 2 As shown, the system includes a physical environment cloud database, a path category cloud database, a mobile terminal cloud database, a video stream cloud database, a mapping cloud database and other databases. Through intelligent calculation and prediction such as through-learning, the current and historical path planning and scheduling result data are obtained. The specific through-learning and other intelligent calculation and prediction methods and the current and historical path planning and scheduling result data are fed back to the mapping cloud database in real time; the mapping cloud database classifies or clusters the video streams by analyzing the spatiotemporality, correlation, difference, content mapping and other issues of large-scale video streams, and statistically calculates various categories of videos. A mapping cloud database is established based on the spatiotemporal mapping method, mapping content, mapping efficiency and mapping accuracy of the flow environment and the physical environment, so that the input physical environment, path category, mobile terminal, video stream, etc. can be matched from the mapping cloud database to the path planning and scheduling related technologies, scenarios, maps, positioning and / or navigation methods, routes, navigation text, voice language, etc.; the current and historical path planning and scheduling result data will be fed back to mobile terminals and other devices in real time to guide various safe movements and / or flights of mobile terminals, and as a reference input, it will help guide the update of the next step of path planning and scheduling related calculations and result data in real time.

[0164] like Figure 3 As shown, according to the actual prior knowledge of the actual physical environment and the monitoring video stream, the specific situation can be obtained, the regularity of the complex dynamic environment in different locations can be analyzed and studied, and a path planning and scheduling video stream intelligent processing network is established. The video stream intelligent processing network is an organic combination of a breadth neural network and a depth neural network for intelligent processing of video streams for path planning and scheduling, forming data and computing resources based on real-time path planning and scheduling of video streams. The video stream intelligent processing network is a computing system and data center for path planning and scheduling that can perform cross-learning intelligent processing of video streams. It can be built in the form of a cloud platform, or it can be built separately for a specific mobile terminal, or the cloud and mobile terminals can be networked together to form a networked system for real-time path planning and scheduling of multiple mobile terminals based on video streams, thereby enhancing flexibility.

[0165] The path planning and scheduling video stream intelligent processing network is divided into storage layer, test layer, training layer and application layer. The training layer is included in the test layer to ensure the consistent and correct availability of data. We will study the theories, models, algorithms, and methods of the test and training layers of the video stream intelligent processing network, study the regularity of the spatial dynamic environment and the regularity of the static environment, and focus on studying the key theoretical and basic technologies of real-time path planning and scheduling based on video streams for complex spatial dynamic environments based on the mutual correlation between the actual physical environment and the video stream environment, the dynamic environment and the static environment, the dynamic environment and the dynamic environment, and the static environment and the static environment. We will study and optimize the theories, models, algorithms, and methods of each layer of the intelligent processing network to form the components of the breadth neural network and deep neural network for intelligent processing of video streams for path planning and scheduling. We will study the various operating composition modes between the components in parallel, serial, cyclic, random, probabilistic, and statistical ways, so that the video stream intelligent processing network becomes a computing system and data center that can perform intelligent processing of video streams for traversal learning for path planning and scheduling, and study and optimize the networked system of real-time path planning and scheduling based on video streams for multiple mobile terminals to provide services for the coordination of path planning and scheduling. Each business service of path planning and scheduling can be regarded as a data unit with a corresponding name, corresponding rules and security processing methods. Data units can share data, share computing, etc., interconnect with each other, and maintain the security, consistency and correctness of data.

[0166] like Figure 4 As shown, the video stream environment of each time period is mapped to the physical environment in time and space, with spatial measurement consistency, and at the same time corresponds to the dynamic environment modeling method and map rapid construction and update algorithm of each time period; Figure 4The spatiotemporal phase mapping and spatial measurement consistency between the video stream environment and the physical environment in time period t3 can be matched and calculated by traversing the learning of the spatiotemporal phase mapping and spatial measurement consistency between the video stream environment and the physical environment in time periods t1 and t2; similarly, the dynamic environment modeling method and the rapid map construction and update algorithm in time period t3 can be matched and calculated by traversing the learning of the dynamic environment modeling method and the rapid map construction and update algorithm in time period t1 and t2, thereby improving the real-time and accuracy of environmental mapping, measurement, modeling and map construction. Figure 4 It can learn through more time periods and perform statistical weighted calculations, etc. Through analyzing the spatiotemporality, relevance, difference, content mapping, etc. of large-scale video streams, and performing correlation analysis between video streams, between frame images, and between video streams and physical environments, the technology has the ability to dynamically trace the spatiotemporal, content positioning, and accurate measurement and judgment of large-scale video streams.

[0167] like Figure 5 As shown, the dynamic environment of the video stream in the t1 time period is mapped with the actual physical environment in time and space, and has the consistency of measurement of dynamic changes in time and space. At the same time, the dynamic characteristics measurement and prediction method of environmental obstacles and / or environmental objects in the t1 time period, the static environment of the video stream in the t2 time period is mapped with the actual physical environment in time and space, and has the consistency of spatial measurement; then Figure 5 In the time period t3, by traversing and learning the spatiotemporal mapping and spatial measurement consistency between the video stream environment and the physical environment in the time period t1 and the time period t2, the spatiotemporal mapping and spatiotemporal measurement consistency between the whole or part of the video stream environment and the whole or part of the actual physical environment required for path planning and scheduling in the time period t3 can be matched and calculated. The video stream environment and the actual physical environment required for path planning and scheduling in each time period may be local or global. The traversal learning technology can learn global or local features and make real-time adaptive adjustments. Similarly, the dynamic characteristic measurement and prediction method of environmental obstacles and / or environmental objects in the time period t3 can be matched and calculated by traversing and learning the dynamic characteristic measurement and prediction method of environmental obstacles and / or environmental objects in the time period t1 or more time periods, thereby improving the real-time and accuracy of dynamic environment mapping, measurement and prediction.

[0168] Similarly, it is possible to learn the relative positioning measurement results and methods of all or part of the environment in the video stream and the dynamics of obstacles and / or related objects and the dynamics of the mobile terminal, the navigation method and navigation route that matches the all or part of the environment in the video stream with the motion characteristics of the mobile terminal, and perform associated positioning and navigation of the dynamics of the environment in the video stream and the dynamics of the mobile terminal, thereby improving the real-time and accuracy of positioning and navigation.

[0169] like Figure 6As shown, according to the actual moving or flying speed, acceleration, trajectory and other conditions of multiple mobile terminals, environmental monitoring video streams, intermediate paths, intermediate target spaces, destinations, etc., the dynamic changes of the network topology are analyzed and predicted, and the real-time positioning of multiple mobile terminals in the network is performed based on node correlation and information routing transmission; the positioning information is combined with the environmental monitoring video stream associated with the location of the mobile terminal and the actual physical environment, and the map is constructed based on the video stream. The positioning navigation is performed based on the environmental dynamics and the mobile terminal dynamics in the associated video stream to continue path planning and scheduling, so that multiple mobile terminals can move or fly safely. At the same time, the navigation information, path planning and scheduling information, and the moving or flying status information of multiple mobile terminals such as speed, acceleration, trajectory, posture, electricity and or oil energy consumption and other status information are fed back to the network input terminal or each mobile terminal input terminal in real time to guide and help the normal operation and optimization of the network path planning and scheduling network system. The dynamic changes of the network topology structure, network node correlation, information routing transmission, and real-time positioning of multiple mobile terminals in the networked system of collaborative path planning and scheduling of multiple mobile terminals are streamed, and the video stream is associated and matched with the corresponding spatial complex dynamic environment monitoring video stream to form the basic environmental elements of "cross-learning" for collaborative path planning and scheduling of multiple mobile terminals, helping and promoting the efficient realization of networked path planning and scheduling of cross-learning. Figure 6 The network, navigation, path planning and scheduling are organically combined to reduce the potential safety hazards in the movement or flight of mobile terminals. Especially in dynamic environment path scheduling, joint research with navigation and network is very necessary. Figure 6 The navigation includes a navigation method that combines video stream and natural language with intelligent processing to improve safety and reliability, but it is not a limitation. This is just an example of an embodiment.

[0170] like Figure 7As shown, the multi-mobile terminal video stream environment in the t1 time period and the multi-mobile terminal physical environment in the t1 time period are used to map the multi-mobile terminal video stream environment in the t1 time period with the actual physical environment in time and space, so that they have the measurement consistency of dynamic changes in time and space. At the same time, claims 1-7 can obtain the measurement, prediction, positioning, and navigation theories, models, algorithms, methods, etc. for multi-mobile terminal collaboration in the t1 time period; the multi-single mobile terminal video stream environment in the t2, t3, … tm multi-time periods and the multi-single mobile terminal physical environment in the t2, t3, … tm multi-time periods are used to map the multi-single mobile terminal video stream environment in the t2, t3, … tm multi-time periods with the corresponding actual physical environment in time and space, so that they have the measurement consistency of dynamic changes in time and space. At the same time, claims 1-7 can obtain The measurement, prediction, positioning, and navigation theories, models, algorithms, and methods of real-time path planning and scheduling of multiple single mobile terminals in multiple time periods t2, t3, ... tm are obtained; the video stream environment of multiple mobile terminals in the tn time period and the physical environment of multiple mobile terminals in the tn time period can be mapped with the time and space of all or part of the actual physical environment according to the needs through the traversal learning of the t1 time period and the t2, t3, ... tm time periods, so that the measurement consistency of dynamic changes in time and space can be achieved. At the same time, the measurement, prediction, positioning, and navigation theories, models, algorithms, and methods of real-time path planning and scheduling of multiple mobile terminals in collaboration can be traversed and learned by traversing the t1 time period and the t2, t3, ... tm time periods. When the environment space is complex and changeable, highly dynamic, and has many uncertain factors, the video stream environment and the actual physical environment that need to be paid attention to in path planning and scheduling in each time period may be local or global. The traversal learning technology can learn global or local features and make real-time adaptive adjustments.

[0171] Figure 8 A schematic diagram of the module connection of a networked path planning and scheduling device for traversal learning based on video streams provided by an embodiment of the present invention. Figure 8 As shown, a video stream-based traversal learning networked path planning and scheduling device includes a configuration module, an acquisition module, an environment modeling module, a map generation module, a navigation module, a path planning and scheduling module, and a visualization module; each module is interconnected through a path planning and scheduling network, and the path planning and scheduling network has a video stream understanding and cognitive system. The video stream intelligent processing network is a component of the video stream understanding and cognitive system. Figure 2 It can be seen that the mapping cloud database is also a component of the video stream understanding and cognitive system. The mapping cloud database can be integrated into the video stream intelligent processing network as a whole, or the mapping cloud database and the video stream intelligent processing network can be interconnected through the path planning and scheduling network.

[0172] A configuration module, the configuration module is used to establish a configuration path planning and scheduling network; an acquisition module, the acquisition module is used to acquire relevant monitoring video streams; an environmental modeling module, the environmental modeling module is used to analyze the internal connections between the four elements of physical environment, monitoring video stream, path category and mobile terminal according to the spatiotemporal phase mapping of the video stream environment and the physical environment and the video stream intelligent processing network, and perform adaptive continuous smooth variable angle dynamic environment modeling for path planning and scheduling; a map generation module, the map generation module is used to build a map in real time based on the spatiotemporal phase mapping of the video stream environment and the physical environment and the video stream intelligent processing network, as well as the data and or results of the environmental modeling module; a navigation module, the navigation module is used to associate the environmental dynamics in the video stream with the dynamics of the mobile terminal for relative positioning navigation; a path planning and scheduling module, the path planning and scheduling module is used to perform adaptive continuous smooth variable angle dynamic environment modeling based on the video stream. Real-time collaborative path planning and scheduling of multiple mobile terminals in a network; a visualization module, which is used to display the data and / or results generated by the configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, the path planning and scheduling module for viewing, and send real-time feedback notifications such as display effects and requirements to the above-mentioned modules, so that the above-mentioned modules can perform real-time adjustment and optimization of algorithms, parameters, models, etc., and re-display the data and / or results; the data and / or results generated by the above-mentioned modules can be displayed in split-screen pages or integrated on the same screen; the configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, the path planning and scheduling module, and the visualization module are interconnected and can share data, transmit data to each other, and provide real-time feedback of information.

[0173] Furthermore, a video stream-based traversal learning networked path planning and scheduling device is characterized in that the configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, the path planning and scheduling module are specifically used to:

[0174] A configuration module, which is used to establish a configuration path planning and scheduling network using cross-learning technology; an acquisition module, which is used to acquire data and information such as current environment monitoring video streams, digital video streams, historical video streams, etc. from mobile terminals, networks, sensors at various locations, etc.; an environment modeling module, which is used to perform real-time environment modeling using cross-learning technology; a map generation module, which is used to build a dynamic map in real time using cross-learning technology; a navigation module, which is used to use cross-learning technology to associate the environmental dynamics and mobile terminal dynamics in the video stream for relative positioning navigation, calculate and predict the relative positioning information and actual physical positioning information of related objects and mobile terminals in the video stream, calculate and predict the moving routes, navigation methods and navigation routes of related objects and mobile terminals in the video stream; a path planning and scheduling module, which is used to use cross-learning technology to perform real-time collaborative path planning and scheduling based on the networking of multiple mobile terminals in the video stream.

[0175] Figure 8The modules of the device are interconnected and interoperable with the path planning and scheduling network to form an example of a networked path planning and scheduling system for crossing learning based on video streaming. The configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, the path planning and scheduling module, and the visualization module are each plug-and-play in the path planning and scheduling network in the form of a device to form a loosely coupled networked path planning and scheduling system for crossing learning based on video streaming. The configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, the path planning and scheduling module, and the visualization module can also be embedded or integrated in the path planning and scheduling network to form an example of a tightly coupled networked path planning and scheduling system for crossing learning based on video streaming. Both the loosely coupled video stream-based traversal learning networked path planning and scheduling system and the tightly coupled video stream-based traversal learning networked path planning and scheduling system can establish and adopt a web operating system, namely a real-time path planning and scheduling web operating system, the visual interface of which is in web form, and the real-time path planning and scheduling web operating system is used to operate and edit the configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, the path planning and scheduling module, the visualization module, etc. of the video stream-based traversal learning networked path planning and scheduling system with corresponding functions, logic, business, rules, models, algorithms, etc.; the real-time path planning and scheduling web operating system can also be used to operate and edit the video stream understanding and cognition system, the video stream intelligent processing network, the mapping cloud database, etc. of the video stream-based traversal learning networked path planning and scheduling system with corresponding functions, permissions, content, logic, business, rules, models, algorithms, etc. Among them, the video stream understanding and cognition system, the video stream intelligent processing network, the mapping cloud database, etc. are also important components of the video stream-based traversal learning networked path planning and scheduling system, and are the efficient data processing, computing and storage centers of the configuration module, the acquisition module, the environment modeling module, the map generation module, the navigation module, the path planning and scheduling module, the visualization module, etc.

[0176] An embodiment of the present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned video stream-based traversal learning networked path planning and scheduling method and device are implemented.

[0177] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0178] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0179] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The above module integration unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0180] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention.

[0181] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0182] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0183] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0184] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0185] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0186] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0187] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A video stream-based traversal learning networked path planning and scheduling method, characterized by: The said cross-learning refers to the theoretical technology and / or method of learning the same and / or different scenes and events and their global and / or local features and related knowledge across time and / or space, and solving problems through intelligent reference, matching, calculation and prediction. The said time is a single or multiple different continuous time periods, the said space is a single or multiple different places or continuous spaces, and the said event is a single or multiple events, wherein the important manifestations of the said event include video streams, voice and text language descriptions, and the said event includes continuously changing events, discretely changing events and continuously and discretely mixed changing events; According to application requirements, the cross-learning is combined with intelligent computing technologies such as mapping learning, deep learning, reinforcement learning, and statistical learning to form a new cross-learning technology, which belongs to multi-source learning technology; The traversal learning networked path planning and scheduling method comprises the following steps: S1: Establish path planning and scheduling network; S2: Obtain relevant monitoring video stream; S3: Build a map based on the video stream; S4: Relative positioning and navigation based on the dynamics of the environment and the mobile terminal in the video stream, including: Analyze the differences in video streams at different time windows, analyze the spatiotemporality of video streams from a global perspective, establish associations between environmental obstacles and / or environmental objects and the actual physical world, and obtain their spatiotemporality and actual physical position and size, density, and weight characteristics in the actual physical world; analyze the differences in multi-associated frame images of video streams, and obtain the relative dynamic changes of local environmental obstacles and / or environmental objects based on the differences and using static and / or dynamic objects or objects as relative references, and measure and predict the dynamic characteristics of obstacles and / or related objects in combination with the actual physical position and characteristics of reference objects or environmental obstacles and / or environmental objects. The dynamic characteristics include: speed, acceleration, moving route, posture, and structural changes. Structural changes include structural changes between various related objects in a certain area, structural changes of a certain object, and structural changes of a certain type of object; Analyze the differences of multiple associated frame images of the video stream, and perform real-time dynamic measurement and prediction of characteristic data of obstacles and / or roads, routes, movable path boundaries, key distances, and key navigation points based on the differences and taking static and / or dynamic things or objects as relative references, and combining the actual physical position and size, density, and weight characteristics of the references; For video streams obtained by the same type of camera sensor, when the sensor parameter settings remain unchanged, the reference object is the relevant object in the obtained historical video stream or the current video stream. When the sensor parameter settings change, the reference object and the relevant object in the current video stream are transformed to the same standard for reference through parameter calculation conversion; For video streams obtained by different types of camera sensors, the reference object and the related object in the current video stream are transformed to the same standard for reference through different types of parameter calculation conversion, and the reference object is the related object in the obtained historical video stream or the current video stream; According to the measurement and prediction of the dynamic characteristics of obstacles and / or related objects, the relative positioning information of the dynamic environment and the dynamic mobile terminal in the video stream is obtained, and the path and navigation route are planned according to the relative positioning information and the road size and destination feature information, so that the mobile terminal can move or fly safely to the target space; S5: Real-time collaborative path planning and scheduling based on video streams for multiple mobile terminals in a network.

2. According to claim 1, a video stream-based traversal learning networked path planning and scheduling method is characterized by: The S1 establishes a path planning and scheduling network, including the following steps: Build network software and hardware infrastructure, including: path planning and scheduling network servers, content, storage, routers and gateways; Establish a route planning and scheduling video stream intelligent processing network: build it separately on the cloud or mobile terminal, or build it together with the cloud and mobile terminals to form a networked system; The networked system is divided into storage layer, test layer, training layer and application layer; The training layer is included in the test layer to ensure the consistent and correct availability of data; The storage layer is used to store video streams and theories, models, algorithms, and data that have passed training and testing; The test layer tests and processes the video stream according to the corresponding services; The training layer performs training processing on the video streams that pass the test according to the corresponding services; The application layer issues service requests and receives service information based on the corresponding services of mobile terminal path planning and scheduling. The service information includes: environment modeling, map construction, route planning, route adjustment, dynamic prediction and specific requirements.

3. According to claim 1, a video stream-based traversal learning networked path planning and scheduling method is characterized by: The S2 acquires relevant monitoring video streams, including: acquiring environmental monitoring video streams, network dynamic topology video streams, and network transmission related video streams; acquiring video streams includes acquiring historical video streams, current video streams, and digital video streams; The acquisition methods include: acquiring from the network, acquiring from the sensor of the mobile terminal camera, acquiring from the sensor of a certain area, acquiring from the relevant sensor of a certain device, and acquiring from the sensor of a certain equipment; The video stream includes: frame images, voice, and text language description related information.

4. The method for networked path planning and scheduling based on video stream traversal learning according to claim 1 is characterized in that: S3 builds a map based on the video stream including: Analyze the internal connections between the physical environment, related monitoring video streams, path categories, and mobile terminals, and build a dynamic map in real time for mobile terminals to move or fly safely to the target space; The dynamic map can be updated in real time according to the changes in the video stream path environment; Dynamic maps include: feasible paths, routes, boundaries, related things and / or objects and descriptions, which include: dimensions, text and symbols; According to the current video stream, the current map is constructed in real time by comparing and matching with the existing digital map of the current area's physical environment and / or digital video stream, historical video stream, and related information; the related information includes: images, voice, text language, and data through identification and reference.

5. The method for networked path planning and scheduling based on video stream traversal learning according to claim 1 is characterized in that: The S4 associates the dynamics of the environment and the dynamics of the mobile terminal in the video stream to perform relative positioning navigation, and further includes: Analyze the spatiotemporal, correlation, difference, and content mapping issues of video streams, classify or cluster video streams, and establish a mapping cloud database based on and statistically calculate the spatiotemporal mapping methods, mapping content, mapping efficiency, and mapping accuracy of various types of video stream environments and physical environments; At the same time, the current physical environment, monitoring video stream, path category, mobile terminal and mapping cloud database are associated to form a path planning and scheduling video stream understanding and cognition system; Based on the understanding and cognitive system, the dynamic characteristics of the current environment obstacles and / or objects are calculated, measured and predicted, the current obstacles and / or objects and the mobile terminal are dynamically relatively positioned, and based on the relative positioning information and target space information, the navigation method and navigation route that match the current mobile terminal motion characteristics are calculated and predicted; According to the spatiotemporal phase mapping between the video stream environment and the physical environment and the video stream intelligent processing network, the internal connections between the four elements of the physical environment, monitoring video streams, path categories and mobile terminals are analyzed, and adaptive continuous smooth variable angle dynamic environment modeling and algorithm optimization for path planning and scheduling are performed, and a map rapid construction and update algorithm for the mobile terminal to safely move or fly to the target space is established; and / or a 360-degree continuous smooth variable angle environment modeling algorithm is established for modeling.

6. The method for networked path planning and scheduling based on video stream traversal learning according to claim 1 is characterized in that: The S5 is based on video stream multi-mobile terminal networking real-time collaborative path planning and scheduling, including: Establish a web-based real-time collaborative route planning and scheduling app and / or related software, including applets and navigation device embedded software; Design the path planning and dispatching network system as a blockchain-based named data network security architecture, combining the naming scheme with the blockchain; Combine naming schemes, semantic information and blockchain to securely transmit video streams in real time; Combine naming schemes, semantic information and privacy protection technology to improve the security and confidentiality of information transmission; Semantic information includes: text, speech, symbols, graphics and images; Analyze and establish a blockchain named data network security architecture for path planning and scheduling based on the logical relationship and structural layout between the path planning and scheduling video stream intelligent processing network and the blockchain-based named data network security architecture and the relevant areas or related spaces where the physical location of the road is located; According to the actual movement or flight status of multiple mobile terminals, environmental monitoring video streams, mid-way paths, mid-way target spaces, and destinations, the dynamic changes of the network topology are predicted. At the same time, based on node correlation and information routing transmission, multiple mobile terminals are networked and positioned in real time to obtain the location of the mobile terminals. The positioning information is combined with the environmental monitoring video stream and the actual physical environment associated with the location of the mobile terminal, and a map is constructed based on the video stream. The dynamics of the environment in the video stream and the dynamics of the mobile terminal are associated to perform positioning navigation and continue path planning and scheduling; The dynamic changes of the network topology structure, network node correlation, information routing transmission, real-time positioning of multiple mobile terminals and related data of the networked system for cooperative path planning and scheduling of multiple mobile terminals are video streamed, and the video stream of the video stream is correlated and matched with the corresponding spatial complex dynamic environment monitoring video stream; Analyze video streams, and study algorithms and collaborative methods for collaborative real-time path planning and scheduling of multiple mobile terminals based on their different characteristics, functions, states, environments, and destinations, using bird flock network topology, ant colony network topology, distributed network topology, centralized network topology, chain network topology, and traffic network network topology; According to the real-time path planning and scheduling demand categories, collaborative network systems and network environments, and video stream intelligent processing network training and testing conditions, collaborative optimization is performed on the real-time path planning and scheduling of multi-mobile terminal networks based on video streams.

7. A video stream-based traversal learning networked path planning and scheduling method according to any one of claims 1 to 6, characterized in that: Use traversal learning technology to establish a path planning and scheduling network; Use traversal learning technology to establish a path planning and scheduling video stream intelligent processing network; Use traversal learning technology for real-time environment modeling; Use cross-learning technology to build maps in real time; Use traversal learning technology to perform relative positioning and navigation based on the dynamics of the environment and the mobile terminal; The traversal learning technology is used to calculate and predict the relative positioning information and actual physical positioning information of relevant objects and mobile terminals in the video stream; Using traversal learning technology to calculate and predict the movement routes, navigation methods and navigation routes of relevant objects and mobile terminals in the video stream; Cross-learning technology is used to perform real-time collaborative path planning and scheduling for multiple mobile terminals in a network.

8. The video stream-based traversal learning networked path planning and scheduling method according to claim 1 is further characterized by: The video stream intelligent processing network is an organic combination of a wide neural network and a deep neural network for intelligent processing of video streams; The theories, models, algorithms, and methods at each layer of the intelligent processing network constitute the components of the breadth neural network and deep neural network for intelligent processing of video streams for path planning and scheduling. The various operating composition modes between the components are studied in parallel, serial, cyclic, random, probabilistic, and statistical ways, making the video stream intelligent processing network a computing system and data center that can perform cross-learning intelligent processing of video streams for path planning and scheduling.

9. A video stream-based traversal learning networked path planning and scheduling system, characterized by: The system can be used to implement the traversal learning networked path planning and scheduling method described in claim 1. The system includes: configuration module, acquisition module, environment modeling module, map generation module, navigation module, path planning and scheduling module and visualization module; The configuration module is used to establish a configuration path planning and scheduling network; The acquisition module is used to acquire relevant monitoring video streams; The environment modeling module is used to analyze the internal connection between the four elements of physical environment, monitoring video stream, path category and mobile terminal according to the spatiotemporal phase mapping between the video stream environment and the physical environment and the video stream intelligent processing network, and to perform adaptive continuous smooth variable angle dynamic environment modeling for path planning and scheduling; The map generation module is used to construct a map in real time based on the spatiotemporal phase mapping of the video stream environment and the physical environment and the video stream intelligent processing network, as well as the data and / or results of the environment modeling module; The navigation module is used to associate the dynamics of the environment and the dynamics of the mobile terminal in the video stream to perform relative positioning navigation; The path planning and scheduling module is used for real-time collaborative path planning and scheduling based on video stream multi-mobile terminal networking; The visualization module is used to display the data and / or results generated by the configuration module, acquisition module, environment modeling module, map generation module, navigation module, path planning and scheduling module for viewing, and send real-time feedback notifications of display effects and requirements to the above modules, so that the above modules can adjust and optimize algorithms, parameters, and models in real time, and display the data and / or results again; the data and / or results generated by the above modules can be displayed in split screens and pages or integrated on the same screen; The configuration module, acquisition module, environment modeling module, map generation module, navigation module, path planning and scheduling module, and visualization module are interconnected and can perform data sharing, data mutual transmission, and real-time mutual feedback of information.

10. The system according to claim 9, characterized in that: The configuration module is used to establish a configuration path planning and scheduling network using traversal learning technology; The acquisition module is used to acquire current environment monitoring video stream, digital video stream, historical video stream data and information from mobile terminals, networks, and sensors at various locations; The environment modeling module is used to perform real-time environment modeling using the traversal learning technology; The map generation module is used to construct a dynamic map in real time using the traversal learning technology; The navigation module is used to use the traversal learning technology to associate the dynamics of the environment and the dynamics of the mobile terminal in the video stream to perform relative positioning navigation, calculate and predict the relative positioning information and actual physical positioning information of the relevant objects and the mobile terminal in the video stream, and calculate and predict the movement route, navigation method and navigation route of the relevant objects and the mobile terminal in the video stream; The path planning and scheduling module is used to utilize the traversal learning technology to perform real-time collaborative path planning and scheduling based on the video stream multi-mobile terminal networking.

11. A networked path planning and scheduling device for traversal learning based on video stream, characterized in that: The device can be used to implement the traversal learning networked path planning and scheduling method described in claim 1, and the device includes: at least one processor, a server and a memory; The memory stores computer-executable instructions; The server establishes a network core node for path planning and scheduling; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs a video stream-based traversal learning networked path planning and scheduling method.

Images