Maritime affair multi-modal data obtaining and updating method

Through Redis block storage and specific data processing strategies, Scrapy's performance bottleneck problem in large-scale data crawling and updating is solved, and efficient update and storage of maritime multimodal data is realized, ensuring real-time and complete data sex.

CN119002800APending Publication Date: 2024-11-22DALIAN MARITIME UNIVERSITY
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410952605.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When large-scale data crawling and updating, the performance of Scrapy combined with array update method may become a bottleneck, especially in the case of high concurrency and large data volume, which may lead to excessive memory usage, slow speed, and data storage in Redis When designing appropriate data storage structures to avoid duplicate data and improve efficiency.

Method used

Redis block storage of maritime multimodal data, and through specific data processing and storage strategies, including using timestamps to represent data update time, traversing the stored identification data to determine data consistency and uniqueness, ensuring timely update of data and Integrity.

Benefits of technology

It realizes efficient data acquisition and update under high concurrency and large data volume, avoids duplicate data, ensures data uniqueness and acquisition efficiency, and ensures the real-time and integrity of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002800A_ABST
    Figure CN119002800A_ABST
Patent Text Reader

Abstract

The invention provides a maritime affair multi-modal data obtaining and updating method. The maritime affair multi-modal data obtaining and updating method comprises the steps that maritime affair multi-modal ship data, port data and news data are obtained; according to the acquired maritime affair multi-modal ship data, updating the multi-modal ship data based on a ship initial; updating the multi-modal port data according to the acquired port data; and updating the multi-modal news data according to the obtained news data. According to the method, the maritime affair multi-modal data are stored by adopting the Redis blocks, the read-write speed based on the Redis blocks is relatively high, repeated data can be screened, and the uniqueness of the data and the efficiency of acquiring the data are ensured. And dynamic data can be introduced as parameters and updated in time, so that the real-time performance of the data is ensured. The method is used in a newly-added file to prevent the situation that all data are not obtained when the data are obtained for the first time, and data integrity is guaranteed. When unstructured data such as ship and port pictures are obtained for the first time, the pictures are obtained by traversing stored ship and port identification data, and the consistency of the data is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of acquiring and updating maritime data, and in particular, to a method for acquiring and updating maritime multimodal data. Background Art

[0002] Requests is a popular HTTP library in Python that simplifies the process of sending HTTP requests and processing responses. Its appearance makes it easier and more convenient to obtain network data in Python.

[0003] With the development of data mining and machine learning technology, a large amount of data is considered to be a valuable resource, so the demand for data acquisition has further increased. Python technology provides a rich source of data for data mining and machine learning.

[0004] With the development of the Internet, the amount of information on the Internet has increased exponentially, and people's demand for obtaining specific data has also increased. Traditional manual methods can no longer meet the growing data needs, so automated network data acquisition technology has become particularly important.

[0005] Scrapy provides a powerful and flexible framework. Before Scrapy, although there were other tools and libraries, they may have some shortcomings in performance, functionality and ease of use. The emergence of Scrapy fills this gap and provides a comprehensive and efficient data acquisition solution. It uses the Twisted asynchronous network framework, which enables it to efficiently send and process a large number of network requests, thereby achieving high concurrency and high efficiency of web page data acquisition. Its design is very modular. It splits the data acquisition process into multiple components, including request sending, page parsing data storage, etc. Each component can be customized and expanded according to needs, allowing developers to flexibly build data acquisition applications according to specific needs. It provides complete data acquisition development tools and processes. Its emergence makes development more standardized and modular, greatly improving development efficiency.

[0006] The Scrapy framework supports the use of XPath and CSS selectors to parse and locate elements on web pages, thereby extracting the required data. These two selectors can flexibly locate elements in web pages and are important tools for data acquisition. It also provides a rich middleware mechanism, including download middleware, middleware, and pipeline middleware, which can process requests and responses at different stages and implement custom functions, such as setting request headers, switching proxies, and cleaning data.

[0007] In the acquisition and update of maritime multimodal data, Scrapy usually temporarily stores the acquired data in memory and processes and stores it through a pipeline mechanism. You can store data in a database, file or other storage medium by writing a custom Pipeline.

[0008] The incremental update method only updates the changed data instead of re-acquiring the entire data set. This can be achieved by comparing the differences between the existing data and the newly acquired data. For example, when acquiring data, important identification information such as the data timestamp is recorded, and then in subsequent acquisitions, data with a timestamp greater than the last crawl is acquired.

[0009] Design appropriate incremental data acquisition strategies, such as incremental data acquisition by time range, incremental data acquisition by page, etc. This can avoid duplicate data acquisition of existing data and improve data acquisition efficiency.

[0010] Formulate data update strategies, including incremental updates and full updates. Incremental updates can be achieved through scheduled tasks or real-time monitoring of data changes to ensure timely data updates; full updates can be performed regularly to ensure data integrity and accuracy.

[0011] As a high-performance key-value storage database, Redis is widely used in scenarios such as data caching and message queues. In Scrapy, the captured data can be stored in Redis, and its fast read and write features can be used to achieve efficient management and update of maritime data.

[0012] By comprehensively utilizing the above technologies, efficient updating of maritime multimodal data based on Scrapy can be achieved, maintaining the timeliness and accuracy of the data.

[0013] The current technical problems are as follows:

[0014] In the current methods for acquiring and updating maritime data, the performance of Scrapy combined with array update methods may become a bottleneck when large-scale data is captured and updated. Especially in the case of high concurrency and large data volumes, it may cause problems such as excessive memory usage, resulting in slow speed. The combination of Scrapy and Redis requires the design of a suitable data storage structure or method to store the cleaned maritime data in Redis, and the design needs to consider the necessary values ​​for data storage. Summary of the invention

[0015] According to the technical problems raised above, a method for acquiring and updating maritime multimodal data is provided. The present invention uses the Scrapy framework to acquire and update maritime data, designs appropriate acquisition structures and rules, and ensures that multimodal data can be effectively acquired from the target website. The acquired data is cleaned and processed, noise data is removed, and data format conversion and standardization are performed for subsequent knowledge graph storage and use. In addition, the present invention can ensure the timely update of data, ensure the integrity and accuracy of data, solve the problem of update speed, and improve the data update method for subsequent data acquisition and update.

[0016] The technical means adopted by the present invention are as follows:

[0017] A method for acquiring and updating maritime multimodal data, comprising:

[0018] Obtain maritime multimodal ship data, port data, and news data;

[0019] According to the acquired maritime multimodal ship data, the multimodal ship data is updated based on the ship's first letter;

[0020] Update multimodal port data based on the acquired port data;

[0021] Update the multimodal news data based on the acquired news data.

[0022] Furthermore, the acquisition of maritime multimodal ship data, port data, and news data specifically includes:

[0023] Obtain maritime multimodal ship data. Basic ship information is structured data, which exists in the form of tables or database records, which is easy to classify and analyze; ship images and ship trajectories are unstructured data, which are processed and analyzed through specific technologies and tools;

[0024] Obtain port data. The basic information, ship arrival time, departure time, and cargo loading and unloading status in the port data are structured data, while the port image data is unstructured data.

[0025] Get news data. The publisher and publishing time in the news data are structured data, while the news content is unstructured data.

[0026] Get the corresponding data dictionary, which exists in the website structured data;

[0027] Observe the data and check whether the website requires login. Since the number of ships displayed on each page of some websites can be adjusted, find the optimal parameters for displaying each page of data according to the time of obtaining data to improve the acquisition speed.

[0028] After successfully obtaining the first page of data, calculate the number of pages of data that need to be obtained, obtain the total number through the first request, and then calculate how many pages of data need to be obtained based on the number of each page;

[0029] The field data related to the ship structured data text update time and the ship construction time are expressed in the form of timestamps. The timestamps are calculated from 00:00:00 on January 1, 1970. The timestamps before this time point are negative numbers, and the timestamps after this time point are positive numbers.

[0030] Further, the updating of the multimodal ship data based on the first letter of the ship according to the acquired maritime multimodal ship data specifically includes:

[0031] The method used to obtain ship structured data for the first time is as follows:

[0032] Traverse the stored ship's newly added identification data, determine whether the ship's Imo number data passed by the parameter exists in the ship's newly added identification data, if not, store the ship's Imo number data passed by the parameter in the ship's newly added identification data, and finally return True or False;

[0033] When you need to store any field representing the ship update, the console will prompt the user to enter the field content of the ship update storage model, separated by spaces, and strictly follow the item class sub-segment naming. Please be sure to store Imo as the first parameter. If there is a time parameter, please change the time structure to something similar;

[0034] If the string entered by the user in the console is not recognized as Imo or imo, an error message will be displayed to the user, indicating that there is a problem with the order or naming of the field identifiers entered in Redis;

[0035] If the number of characters in the string entered by the user in the console is inconsistent with the parameters passed by the method, all stored data will be deleted and an error will be reported. The error content is that the field length provided by the console is inconsistent with the length of the passed parameter; the console provides specific data of the field length and the specific data of the length of the passed parameter, and displays the content "Sorry! Please re-store all data";

[0036] If there is no error in any of the above, the update identification field corresponding to the ship's Imo number is stored in the ship update identification data;

[0037] The method used to obtain ship image data for the first time is as follows:

[0038] Traverse the stored ship's newly added identification data. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence";

[0039] If there are no errors above, it will determine whether the Imo number passed from this page has been stored, and return True or False;

[0040] The method used by the newly added file of ship structured data is as follows:

[0041] Traverse the stored new ship identification data to determine whether the Imo number passed from this page has been stored. If it has not been stored, store the first letter of the newly added ship, store the Imo number of this ship in the new ship identification data or store it in the new identification data of the newly added ship, and store the first letter of the ship in the new ship first letter data;

[0042] Use the Imo number passed from the page to determine whether it has been stored before as a mark and return True or False;

[0043] When a new ship appears, the console will prompt the user, prompting "Please enter the length and order of the field content of the ship storage model in the new file and the field content stored in Redis. Please be sure to store Imo as the first parameter. If there is a time parameter, please change the time structure to be similar";

[0044] When the user enters other letters, the system will continue to allow the user to enter;

[0045] When the user enters N / n, all stored data will be deleted and an error will be reported, with the error content being "Sorry! Please re-store all data";

[0046] When the number of field parameters passed is inconsistent with the number of storage memory fields, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0047] Traverse the stored ship's newly added identification data. If there is no data, an error will be reported. The error content is "The execution file sequence or empty data is wrong. Please execute the file in sequence";

[0048] If there is no error in any of the above, the update identification field corresponding to the ship's Imo number will be stored in the ship's update identification data;

[0049] The method used to obtain the newly added ship image data is as follows:

[0050] Traverse the stored data of the newly added ship's first letters. If there is data, return all the data; if there is no data, report an error with the error content "There is currently no new ship's first letter information";

[0051] Traverse and store the newly added identification data of the newly added ship. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong, please execute the file in sequence";

[0052] Determines whether the ship IMO number passed from the page is in the newly added identification data of the newly added ship, and will return True or False;

[0053] The method used to update the ship structured data file is as follows:

[0054] Traverse the stored ship update identification data. If it is a null value, an error is reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence";

[0055] When the number of field parameters passed is inconsistent with the number of storage memory fields, an error will be reported, and the error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Please modify it and re-run this file."

[0056] When a ship with updated data appears, the console will prompt the user, with the prompt content "Please enter the length and order of the field content of the ship storage model in the update file and the field content stored in Redis. Please be sure to store Imo as the first parameter. If there is a time parameter, please change the time structure to be similar";

[0057] When the user enters N / n in the console, an error message will be displayed, saying "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful."

[0058] If there is no error in any of the above, based on the same ship Imo number, compare the ship update identification field value passed from the page with the stored update identification field value to see if they are the same. If they are different, update the update identification field data corresponding to the stored ship Imo number;

[0059] Returns the record flag of whether the ship data passed from the page needs to be updated, returning True or False;

[0060] When the user obtains the newly added ship data for the second time or multiple times, the newly added identification data of the newly added ship needs to be deleted.

[0061] Further, the updating of the multimodal port data according to the acquired port data specifically includes:

[0062] The method used to obtain port structured data for the first time is as follows:

[0063] Traverse the stored port new identification data, determine whether the port UN_Locode, Name, Country data passed by the page has been stored, if not, store the passed port UN_Locode, Name, Country data in the port new identification data, and finally return True or False;

[0064] When a port needs to be stored in the port update identification data, the console will prompt the user, prompting "Please enter the field content of the port update storage model, separated by spaces, and strictly follow the item class field naming. Please be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter";

[0065] If the user enters parameters in a different order in the console, an error message will be displayed, saying "There is a problem with the order or naming of the field identifiers entered in Redis";

[0066] If the number of characters in the string entered by the user in the console is inconsistent with the parameters passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0067] If there is no error in any of the above, the update identification field corresponding to the port is stored in the update identification data;

[0068] The method used to obtain port image data for the first time is as follows:

[0069] Traverse the stored port new identification data. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence";

[0070] If there is no error, determine whether the UN_Locode, Name, Country data passed by the page has been stored, and return True or False;

[0071] The method used by the newly added file of port structured data is as follows:

[0072] Traverse the newly added identification data of the port to determine whether the port UN_Locode, Name, Country data passed by this page has been stored. If not, store the port UN_Locode, Name, Country data passed by the page in the newly added identification data of the port or in the newly added identification data of the newly added port;

[0073] Returns whether the port UN_Locode, Name, Country data passed from the page has been stored as a mark, and returns True or False;

[0074] When a new port appears, the console will prompt the user, prompting "Please enter the length and order of the field content of the port storage model in the new file and the field content stored in Redis. Please be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter (Y / N)";

[0075] When the user enters other letters, the system will continue to allow the user to enter;

[0076] When the user enters N / n, all stored data will be deleted and an error message will be displayed, with the message "Sorry! Please re-store all data";

[0077] When the number of field parameters passed is inconsistent with the number of storage memory fields, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0078] Traverse the port to update the identification data. If there is no data, an error will be reported. The error content is "The execution file sequence or empty data is wrong. Please execute the file in sequence";

[0079] If there is no one of the above errors, the port update identification field passed by the parameter is stored in the update identification data;

[0080] Returns whether the port update identification field passed from the page is a newly added port record flag, and returns True or False;

[0081] The method used to obtain the newly added port image data is as follows:

[0082] Traverse the newly added identification data of the newly added port. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence";

[0083] Determines whether the port UN_Locode, Name, Country data passed from the page is in the newly added identification data of the newly added port, and returns True or False;

[0084] The method used to update the port structured data file is as follows:

[0085] Traverse the port to update the identification data. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence";

[0086] When the number of field parameters passed is inconsistent with the number of storage memory fields, an error will be reported, and the error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Please modify it and re-run this file."

[0087] When a port needs to update data, the console will prompt the user, prompting "Please enter the length and order of the field content of the port storage model in the newly added file and the field content stored in Redis. Please be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter (Y / N)";

[0088] When the user enters N / n in the console, an error message will be displayed, saying "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful."

[0089] If there is no error in any of the above, based on the same port UN_Locode, Name, Country data, compare the port update identification field value passed from the page with the stored update identification field value to see if they are the same. If they are different, update the corresponding update identification field data of the stored port;

[0090] Returns whether the port data passed from the page is the record flag of the updated port, returns True or False;

[0091] When the user obtains the newly added port data for the second time or multiple times, it is necessary to delete the stored newly added identification data of the newly added port.

[0092] Further, the updating of the multimodal news data according to the acquired news data specifically includes:

[0093] The method used to obtain news for the first time is as follows:

[0094] If the date format of the passed parameter is incorrect, an error message will be displayed, with the message "Date storage error, please recheck and run this file";

[0095] When there is news content that needs to be stored, the console will prompt the user, and the prompt content is "Please enter the field content of the news update storage model, separated by spaces, and pass it strictly according to the method parameters. The first parameter is preferably the ship's Imo number as the identifier. If the Imo number is empty, the ship's Name can be passed as the identifier; if this website has no news category, the second parameter passes an empty value. Please change the time parameter structure to something similar";

[0096] If the first parameter entered by the user in the console is not Imo or Name, an error message will be displayed to the user, with the error content being "There is a problem with the order or name of the field identifiers entered in Redis";

[0097] If the number of characters in the string entered by the user in the console is inconsistent with the parameters passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0098] If there is no error in any of the above, the four fields of the transmitted news data, including the news content, are converted into ASCII code and stored in the news update identification data;

[0099] The method used to obtain news user pictures for the first time is as follows:

[0100] If the date format of the passed parameter is incorrect, an error message will be displayed, with the message "Date storage error, please recheck and run this file";

[0101] Traverse the stored news update identification data. If it is a null value, an error is reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence";

[0102] Determine whether the four parameter field data passed from the page have been stored in the news update identification data, and return the record flag and True or False, wherein the news content data of the passed parameters is converted into ASCII code;

[0103] Traverse the stored news user picture Url data, determine whether the news user picture Url passed from the page has been stored, if not, store the news user picture Url in the news user picture Url data, and return the record flag, True or False;

[0104] The method used by the news update file is as follows:

[0105] If the date format of the passed parameter is incorrect, an error message will be displayed, with the message "Date storage error, please recheck and run this file";

[0106] Traverse the stored news update identification data. If it is a null value, an error is reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence";

[0107] If the number of characters in the string entered by the user in the console is inconsistent with the parameters passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0108] If there are no errors above, the console will prompt the user, and the prompt content is "Please enter the field content of the news update storage model, separated by spaces, and pass it strictly according to the method parameters. The first parameter is preferably the ship's Imo number as the identifier. If the Imo number is empty, the ship's Name can be passed as the identifier; if this website has no news category, the second parameter passes an empty value. Please change the time parameter structure to something similar";

[0109] When the user enters N / n in the console, an error message will be displayed, saying "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful."

[0110] If there are no errors in the above, the passed news content parameter value is converted into Ascii code, and the passed data is judged whether the ship Imo or Name parameter field data value passed from the page is in the stored news update identification data. If it has not been stored, the news data is stored in the news update identification data and stored in the updated news update identification data; if it has been stored, it is judged whether the news category parameter field data value passed by the same ship is in the stored news update identification data. If it has not been stored, the news data is stored in the news update identification data and stored in the updated news update identification data; if it has been stored, under the conditions of the same ship and the same news category, the format of the passed news release time parameter field data value is judged and converted into a timestamp and compared with the time extracted from the stored news update identification data. If the time passed by the parameter is longer than the stored time, then the news is saved in the news update identification data. The stored news time is updated, the news data is stored in the news update identification data, and stored in the updated news update identification data; if the time is older, no processing is performed; if the time is equal, it is determined whether the ASCII code converted from the passed news content parameter value is the same as the ASCII code of the ship news stored in the news update identification data. If the news is the same, no processing is performed; if the news is not the same, the news data updated on that day is traversed and stored. If the news passed by the parameter is not in the news data updated on that day, all news of the same category of this ship are deleted, the update identification data of this ship news is updated, and the ASCII code converted from the passed news content parameter value is stored in the updated news data; if the ASCII code converted from the passed news content parameter value is in the updated news data, no processing is performed;

[0111] Returns the record flag of whether the news parameter value passed from the page needs to be updated, and returns True or False;

[0112] The method used for news user picture update files is as follows:

[0113] If the date format of the passed parameter is incorrect, an error message will be displayed, with the message "Date storage error, please recheck and run this file";

[0114] Traverse the updated news update identification data. If it is a null value, an error is reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence";

[0115] Convert the passed news parameter value into ASCII code, determine whether the four parameter values ​​passed from the page are stored in the updated news update identification data, and return the record flag, True or False;

[0116] Traverse the news user picture Url data, determine whether the news user picture Url data passed from the page has been stored, and return the record flag, True or False;

[0117] When the user obtains updated news data for the second time or multiple times, it is necessary to delete the stored updated news update identification data.

[0118] Compared with the prior art, the present invention has the following advantages:

[0119] 1. The present invention provides a method for acquiring and updating maritime multimodal data, which adopts Redis block storage for maritime multimodal data. Some website data may have the same situation. The speed of reading and writing based on Redis block is relatively fast. The method of the present invention can filter out duplicate data, that is, ensure the uniqueness of data and the efficiency of obtaining data.

[0120] 2. The method for acquiring and updating maritime multimodal data provided by the present invention can input dynamic data as parameters and update them in time, thus ensuring the real-time nature of the data.

[0121] 3. The method for acquiring and updating maritime multimodal data provided by the present invention is used in a newly added file to prevent the situation where all data are not acquired for the first time, that is, to ensure the integrity of the data.

[0122] 4. The present invention provides a method for acquiring and updating maritime multimodal data. When acquiring unstructured data such as ship and port pictures for the first time, the pictures are acquired by traversing the stored ship and port identification data, thereby ensuring data consistency.

[0123] 5. The method for acquiring and updating maritime multimodal data provided by the present invention is universal on most websites. As long as there are ship Imo number and name data, port Un_Locode, name, country data, news release time, news content, and associated ship Imo number or ship name data, the method of the present invention can be used, which ensures the universality of the method of the present invention.

[0124] Based on the above reasons, the present invention can be widely promoted in the fields of acquisition and update of maritime data. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0126] Figure 1 The figure is a flow chart of the method of the present invention.

[0127] Figure 2 This is a flow chart of updating multimodal ship data based on the ship's initials according to the present invention.

[0128] Figure 3 This is a flow chart of the multimodal port data update of the present invention.

[0129] Figure 4 This is a flow chart of updating multimodal news data of the present invention.

[0130] Figure 5 The optimal parameters for testing the local CONCURRENT_REQUESTS provided in the embodiment of the present invention.

[0131] Figure 6 The experimental results of the number of displayed items per page provided by the embodiment of the present invention.

[0132] Figure 7 The three methods provided in the embodiments of the present invention are used to test the time used for the ship structured update file. DETAILED DESCRIPTION

[0133] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0134] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0135] like Figure 1 As shown, the present invention provides a method for acquiring and updating maritime multimodal data, comprising:

[0136] S1. Obtain maritime multimodal ship data, port data, and news data;

[0137] S2. updating the multimodal ship data based on the first letter of the ship according to the acquired maritime multimodal ship data;

[0138] S3. Update the multimodal port data according to the acquired port data;

[0139] S4. Update the multimodal news data based on the acquired news data.

[0140] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:

[0141] S11. Obtain maritime multimodal ship data. Basic ship information is structured data, which exists in the form of tables or database records, which is convenient for classification and analysis; ship images and ship tracks are unstructured data, which are not easy to represent with traditional tables or database records and need to be processed and analyzed through specific technologies and tools;

[0142] S12. Obtain port data. The basic information, ship arrival time, departure time, and cargo loading and unloading conditions in the port data are structured data, and the port image data are unstructured data.

[0143] S13, obtaining news data. The publishing author and publishing time in the news data are structured data, while the news content belongs to unstructured data;

[0144] S14, obtaining a corresponding data dictionary, where the data dictionary exists in the website structured data;

[0145] S15. Observe the data and check whether the website needs to be logged in. Since the number of ships displayed on each page of some websites can be adjusted, find out the optimal parameters for displaying each page of data according to the time of obtaining data to improve the acquisition speed.

[0146] S16. After successfully acquiring the first page of data, calculate the number of pages of data that need to be acquired, obtain the total number through the first request, and then calculate how many pages of data need to be acquired based on the number of each page; to implement the update algorithm, this cannot be treated as a fixed value.

[0147] S17, using timestamp to represent the field data of time related to ship structured data text update time (Update) and ship construction time (Year_build), some ship timestamps are positive, some are negative. For data integrity, it is necessary to process, and the processing method is to calculate the timestamp from 00:00:00 on January 1, 1970. The timestamp before this time point is negative, and the timestamp after this time point is positive. In this embodiment, if you want to obtain a negative timestamp, start from the base time of 1970.

[0148] In this embodiment, the present invention adopts a method of obtaining maritime multimodal ship, port, and news data, wherein ships and ports are important components of the maritime field, and their operating conditions and related events have an important impact on the maritime industry. Ships usually dock at ports for operations such as loading and unloading cargo and refueling, so there is a docking relationship between ship data and port data. By analyzing the docking of ships at different ports, the transportation routes and cargo flows of ships can be understood. News reports usually involve events in the maritime field such as ship accidents and port events. By analyzing news data, the performance and impact of ships and ports in news events can be understood, so as to better understand the operation of ships and ports. Therefore, by comprehensively analyzing the close relationship between ships, ports and news data, by comprehensively analyzing these data, the operation of the maritime industry and related events can be better understood, providing data support for research and application in the maritime field.

[0149] When specifically implemented, as a preferred embodiment of the present invention, Figure 2 As shown, step S2 specifically includes:

[0150] S21. The method used when acquiring ship structured data for the first time is as follows:

[0151] Add_ship_data_to_keys_with_increased_fields_using_Redis method (parameters: website name, ship first letter, Imo number):

[0152] S211, traverse the stored ship's newly added identification (Imo number) data, determine whether the ship's Imo number data passed by the parameter exists in the ship's newly added identification (Imo number) data, if not, store the ship's Imo number data passed by the parameter in the ship's newly added identification (Imo number) data, and finally return True or False;

[0153] Add_ship_data_to_keys_with_updated_fields_using_Redis method (parameters: website name, ship first letter, any field):

[0154] S212. When any field representing the ship update needs to be stored, the console will prompt the user to enter the field content of the ship update storage model, separated by spaces (please strictly follow the item class sub-segment naming, and be sure to store Imo as the first parameter). If there is a time parameter, please change the time structure to something like; (%Y-%m-%d %H:%M:%S)

[0155] S213. If the string entered by the user in the console is not recognized as Imo or imo, an error message will be displayed to the user, indicating that there is a problem with the order or naming of the field identifiers entered in Redis.

[0156] S214. If the number in the string input by the user in the console is inconsistent with the parameter passed by the method, all stored data will be deleted and an error will be reported. The error content is that the field length provided by the console is inconsistent with the length of the passed parameter; the console provides specific data of the field length and the specific data of the length of the passed parameter, and displays the content "Sorry! Please re-store all data";

[0157] S215. If there is no error in any of the above, the update identification field corresponding to the ship Imo number is stored in the ship update identification data;

[0158] S22. The method used when acquiring ship image data for the first time is as follows:

[0159] Read_the_stored_ship_Imo_number_data_and_identify method (parameters: website name, ship first letter, Imo number):

[0160] S221, traverse the stored ship newly added identification data, if it is a null value, an error is reported, and the error content is "execution file sequence or null data error, please execute the file in sequence";

[0161] S222. If there is no error above, it will determine whether the IMO number passed from this page has been stored, and return True or False;

[0162] S23. The method of using the newly added file of ship structured data is as follows:

[0163] Use_Redis_to_identify_newly_added_ships_and_add_them_to_the_key method (parameters: website name, ship first letter, Imo number):

[0164] S231, traverse the stored newly added identification data of the ship, determine whether the Imo number transmitted from this page has been stored, if not stored, store the first letter of the newly added ship, store the Imo number of the ship in the newly added identification data of the ship or store it in the newly added identification data of the newly added ship, and store the first letter of the ship in the newly added ship first letter data;

[0165] S232, whether the Imo number passed from the page has been stored is used as a mark, and True or False is returned;

[0166] Add_new_ship_data_to_keys_with_updated_fields_using_Redis method (parameters: website name, ship first letter, any field):

[0167] S233. When a new ship appears, the console will prompt the user, and the prompt content is "Please enter the length and order of the field content of the ship storage model in the new file and the field content stored in Redis. Please be sure to store Imo as the first parameter. If there is a time parameter, please change the time structure to be similar"; (%Y-%m-%d %H:%M:%S) (Y / N)

[0168] S234, when the user inputs other letters, the user will continue to input;

[0169] S235. When the user enters N / n, all stored data will be deleted and an error message will be given, with the error message content being “Sorry! Please re-store all data”;

[0170] S236. When the number of field parameters passed is inconsistent with the number of storage memory fields, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0171] S237, traverse the stored ship's newly added identification data, if there is no data, an error is reported, and the error content is "execution file sequence or empty data error, please execute the file in sequence";

[0172] S238. If there is no error in any of the above, the update identification field corresponding to the ship Imo number is stored in the ship update identification data;

[0173] S24. The method used to obtain the newly added ship image data is as follows:

[0174] Read_new_ship_letter_information method (parameter: website name):

[0175] S241, traverse the stored data of the first letter of the newly added ship, and if there is data, return all the data; if there is no data, report an error, and the error content is "there is currently no first letter information of the newly added ship";

[0176] Read_the_new_ship_storage_key_and_determine_if_the_ship_is_newly_added method (parameters: website name, ship first letter, IMO number):

[0177] S242, traverse and store the newly added identification data of the newly added ship, if it is a null value, report an error, the error content of which is "execution file sequence or null data error, please execute the file in sequence";

[0178] S243, judging whether the ship IMO number transmitted from the page is in the newly added identification data of the newly added ship, and returning True or False;

[0179] S25. The method for updating the ship structured data file is as follows:

[0180] Determine_Whether_The_Ship_Needs_To_Update_And_Modify_Stored_Data method (parameters: site name, ship first letter, any field):

[0181] S251, traverse the stored ship update identification data, if it is a null value, report an error, the error content of which is "execution file sequence or null data error, please execute the file in sequence";

[0182] S252. When the number of field parameters passed is inconsistent with the number of storage memory fields, an error will be reported, and the error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Please modify it and re-run this file."

[0183] S253. When a ship with updated data appears, the console will prompt the user, and the prompt content is "Please enter the length and order of the field content of the ship storage model in the update file to ensure that it is consistent with the field content stored in Redis. Please be sure to store Imo as the first parameter. If there is a time parameter, please change the time structure to something like (%Y-%m-%d %H:%M:%S) (Y / N)";

[0184] S254. When the user enters N / n in the console, an error message will be displayed, which reads "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful."

[0185] S255. If there is no error in any of the above, based on the same ship IMO number, compare the ship update identification field value transmitted from the page with the stored update identification field value to see if they are the same. If they are different, update the update identification field data corresponding to the stored ship IMO number;

[0186] S256, returning a record flag indicating whether the ship data passed from the page needs to be updated, returning True or False;

[0187] S257. When the user obtains the newly added ship data for the second time or multiple times, the newly added identification data of the newly added ship needs to be deleted. The ClearTheKeyForAddingAndUpdatingNewShips method is executed.

[0188] In this embodiment, the Imo number of the ship is its unique identifier. When the ship structured data is obtained for the first time, by storing the Imo number, the non-existent ship can be quickly found and a new ship can be added. By storing the update identification field data, the ship that needs to be updated can be quickly found. In the ship structured data new file, the model determines whether the current Imo number is newly added. If it is newly added, the newly added Imo number is stored in the newly added identification data and a new memory space is opened to store the newly added Imo number, and the first letter of the newly added ship is stored in the newly added ship letter data. Finally, the ship Imo number and the update identification field are stored in the update identification data. In the ship structured data update file, the model is based on the same ship Imo number. When other fields obtained from the update file are inconsistent with the stored field data, the field for this ship is modified. Then delete the corresponding storage record, update the latest Imo number and other field storage records. The characteristic of some websites is that when the update time of the structured data changes, it means that the basic data of the ship has changed. The update method can be implemented by passing two parameters, namely the Imo number and the update time, into the model. Under the premise of the same Imo number, if the ship update time obtained from the update file is inconsistent with the stored update time, it is considered that the ship has a field modified. Then delete the corresponding storage record, and update the latest Imo number and the corresponding data update time storage record. The characteristic of some websites is that the time of ship structured data refers to (Last_Seen) changes, and the basic information of the ship has not changed. For such data, the model needs to meet the requirements that each field of the ship can be obtained when it changes, so as to improve the integrity of the data. Based on the data with the above two characteristics, a model method for updating multimodal data of ships with arbitrary parameters can be implemented.

[0189] The present invention obtains the fastest method by verifying the storage time of different storage methods. When implementing the ship update file, the time of the following three different methods was tested. Selecting a suitable data storage method is crucial to the performance and efficiency of the system. It aims to compare the efficiency of the update method of array, Redis and Redis block storage data to determine which method is faster and more efficient under certain specific conditions. A test set containing a large amount of ship data is used to ensure that the performance in various situations can be fully reflected. Experiments are carried out under the same hardware and software environment to eliminate the influence of external factors on the results. The update method is implemented on the array, Redis and Redis block storage data respectively, and the execution time is recorded. As we all know, when crawling and updating small-scale data, you can choose to implement the update method based on the array, which is usually more intuitive and simple, easy to understand and implement, and can reduce unnecessary complexity. It is a data structure for continuous storage, and the use of memory is more efficient. Scrapy combined with Redis can realize distributed management, multiple nodes can share the same Redis database, realize the distribution and scheduling of task queues, thereby improving the efficiency and speed of data acquisition. Redis's data structure supports functions such as task queues and collections. It can be used to store URL sets to be retrieved and URL sets that have been retrieved to implement task scheduling and deduplication. This ensures that data acquisition of pages that have already been captured will not be repeated, improving the efficiency and performance of acquisition. Redis supports persistent storage, and the acquired data can be temporarily stored in Redis to prevent data loss or duplicate data acquisition. This is very helpful for data processing and subsequent analysis. In addition, it can be used as a cache database to store acquired data or processed data, thereby accelerating data access and acquisition. This is very useful for large-scale data acquisition and processing. Redis can be used to monitor the operating status, task progress, and performance indicators of the acquisition in real time, so as to promptly discover and solve problems and ensure the stable operation of the acquisition.

[0190] Use Redis to store key-value pairs in blocks. By dividing the key-value pairs into multiple blocks and storing them in Redis, a large amount of data can be dispersed into different partitions, thereby reducing the amount of data in a single partition and reducing the memory pressure of a single Redis instance. When Redis stores data with a single key, the ship's Imo number can be stored in the same key. When Redis stores data in blocks, since some websites are sorted by the first letters A to Z, the stored keys can be divided into blocks according to A to Z. It is necessary to loop to obtain the ship data and the total number of ship pages in each English letter. If there is no ship information website sorted by the first letters A to Z, based on this data format that is similar to most websites, you can get the first letter of its name and then store it in blocks. To achieve the acquisition of unstructured ship image data: 1. Use the browser developer tool to analyze the website structure of the ship provider's image to find out the page, element and URL where the target data is located; 2. In order to facilitate the subsequent processing and integration of images, the file name must adopt a unique identifier, so it is also necessary to obtain the ship's Imo number; 3. When acquiring the ship image data for the first time, traverse the stored ship Imo number data to determine whether the Imo number obtained on the page is stored. When acquiring new ship image data, read the first letter of the stored new ship, and traverse the stored new ship Imo number data to determine whether the Imo number obtained on the page is stored. This can not only improve the speed, but also ensure the consistency of the data stored in the two files.

[0191] When specifically implemented, as a preferred embodiment of the present invention, Figure 3 As shown, step S3 specifically includes:

[0192] S31. The method used when obtaining port structured data for the first time is as follows:

[0193] Add_port_data_to_keys_with_increased_fields_using_Redis method (parameters: website name, UN_Locode, Name, Country):

[0194] S311, traverse the stored port newly added identification data, determine whether the port UN_Locode, Name, Country data transmitted by the page has been stored, if not, store the transmitted port UN_Locode, Name, Country data in the port newly added identification data, and finally return True or False;

[0195] Add_port_data_to_keys_with_updated_fields_using_Redis method (parameters: website name, any field):

[0196] S312. When a port needs to be stored in the port update identification data, the console will prompt the user, and the prompt content is "Please enter the field content of the port update storage model, separated by spaces (please strictly follow the item class field naming, and be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter)".

[0197] S313. If the user enters parameters in a different order in the console, an error message will be displayed, which reads "There is a problem with the order or naming of the field identifiers entered in Redis";

[0198] S314. If the number in the string entered by the user in the console is inconsistent with the parameter passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0199] S315. If there is no error in any of the above, the update identification field corresponding to the port is stored in the update identification data;

[0200] S32. The method used when obtaining port image data for the first time is as follows:

[0201] Read_the_stored_port_data_and_identify method (parameters: website name, UN_Locode, Name, Country):

[0202] S321, traverse the stored port newly added identification data, if it is a null value, report an error, the error content of which is "the execution file sequence or null data is wrong, please execute the file in sequence";

[0203] S322. If there is no error, determine whether the UN_Locode, Name, Country data passed by the page has been stored, and return True or False;

[0204] S33. The method of using the newly added file of port structured data is as follows:

[0205] Use_Redis_to_identify_newly_added_port_and_add_them_to_the_key method (parameters: website name, UN_Locode, Name, Country):

[0206] S331, traverse the newly added identification data of the port, determine whether the port UN_Locode, Name, Country data transmitted by this page has been stored, if not stored, store the port UN_Locode, Name, Country data transmitted by the page into the newly added identification data of the port or into the newly added identification data of the newly added port;

[0207] S332, returns whether the port UN_Locode, Name, Country data passed from the page has been stored as a mark, and returns True or False;

[0208] Add_new_port_data_to_keys_with_updated_fields_using_Redis method (parameters: website name, any field):

[0209] S333. When a new port appears, the console will prompt the user, and the prompt content is "Please enter the length and order of the field content of the port storage model in the new file to ensure that it is consistent with the field content stored in Redis. Please be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter (Y / N)";

[0210] S334, when the user inputs other letters, the user will continue to input;

[0211] S335. When the user enters N / n, all stored data is deleted and an error message is given, which reads "Sorry! Please re-store all data."

[0212] S336. When the number of field parameters passed is inconsistent with the number of storage memory fields, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0213] S337, traverse the port update identification data, if there is no data, an error is reported, and the error content is "execution file sequence or empty data error, please execute the file in sequence";

[0214] S338. If there is no error in any of the above, the port update identification field passed by the parameter is stored in the update identification data;

[0215] S339, returning whether the port update identification field passed from the page is a newly added port record flag, and returning True or False;

[0216] S34. The method used to obtain the newly added port image data is as follows:

[0217] Read_the_new_port_storage_key_and_determine_if_the_port_is_newly_added method (parameters: website name, UN_Locode, Name, Country):

[0218] S341, traverse the newly added identification data of the newly added port, if it is a null value, an error is reported, and the error content is "the execution file sequence or null data is wrong, please execute the file in sequence";

[0219] S342, determine whether the port UN_Locode, Name, Country data passed from the page is in the newly added identification data of the newly added port, and return True or False;

[0220] S35. The method used for updating the port structured data file is as follows:

[0221] Determine_Whether_The_Port_Needs_To_Update_And_Modify_Stored_Data method (parameters: website name, any field):

[0222] S351, traverse the port update identification data, if it is a null value, an error is reported, and the error content is "the execution file sequence or null data is wrong, please execute the file in sequence";

[0223] S352. When the number of field parameters passed is inconsistent with the number of storage memory fields, an error will be reported, and the error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Please modify it and re-run this file."

[0224] S353. When a port needs to update data, the console will prompt the user, and the prompt content is "Please enter the length and order of the field content of the port storage model in the newly added file and the field content stored in Redis. Please be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter (Y / N)";

[0225] S354. When the user enters N / n in the console, an error message will be displayed, which reads "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful."

[0226] S355. If there is no error in any of the above, based on the same port UN_Locode, Name, Country data, compare the port update identification field value passed from the page with the stored update identification field value to see if they are the same. If they are different, update the stored update identification field data corresponding to the port;

[0227] S356, returning whether the port data passed from the page is a record flag of the updated port, returning True or False;

[0228] S357: When the user obtains the newly added port data for the second time or multiple times, it is necessary to delete the stored newly added identification data of the newly added port and execute the ClearTheKeyForAddingAndUpdatingNewPorts method.

[0229] In this embodiment, the port structured data field can be divided into dynamic data and static data. When the data is updated, the dynamic data is updated frequently. These data also provide dynamic data support for the subsequent knowledge graph. The port's UN Locode is its unique identifier. After the port structured data is obtained for the first time, by storing the UN Locode, you can quickly find the non-existent ports and add new ports. Storing the update identification field can quickly find the ports that need to be updated. When obtaining port data, since the UN Locode values ​​of some small ports on some websites are empty, in order to reflect the versatility of the model, the judgment of the empty value is designed in the model method. If it is empty, it is necessary to judge the name and country, and realize the storage of the three fields of UNLocode, Name, and Country. In the port structured data new file, the model determines whether the current port is newly added. If so, the newly added identification field data of the newly added port is stored, and then a new memory is opened to store the newly added identification field data of the newly added port, and finally the update identification field data is stored in the memory. Storing the port update identification field data can realize the update method, so that the data can be updated and obtained when each field of the port changes. Some of the fields are dynamic data, which can ensure the integrity and real-time nature of their data. Under the premise of the same port UN Locode, Name, and Country, when other fields obtained from the update file are inconsistent with the stored field data, it is considered that the port has a field modified. Then delete the corresponding update identification field data and update the latest port update identification field data. To achieve the acquisition of unstructured port image data: 1. Use the browser developer tool to analyze the website structure and find out the page, element, and URL where the target data is located; 2. In order to facilitate subsequent image processing and integration, the file name must adopt a unique identifier, but because UN Locode cannot guarantee that all values ​​exist, when its value is empty, it is named with the port name; 3. When acquiring the port image data for the first time, traverse the stored port UN Locode, Name, and Country data. When acquiring the newly added port image data, traverse the stored newly added port data to determine whether the port acquired on the page is stored to ensure the consistency of the data stored in the two files.

[0230] When specifically implemented, as a preferred embodiment of the present invention, Figure 4 As shown, step S4 specifically includes:

[0231] S41. The method used when obtaining news for the first time is as follows:

[0232] Add_News_data_to_keys_with_updated_fields_using_Redis method (parameters: website name, ship IMO number or name, news category, release time, news content):

[0233] S411. If the date format of the passed parameter is wrong, an error will be reported, and the error content is "Date storage error, please recheck and run this file";

[0234] S412. When there is news content to be stored, the console will prompt the user, and the prompt content is "Please enter the field content of the news update storage model, separated by spaces, and please strictly pass according to the method parameters. The first parameter is preferably the ship's Imo number as the identifier. If the Imo number is empty, the ship's Name can be passed as the identifier; if this website has no news category, the second parameter passes a null value. Please change the time parameter structure to something like (%Y-%m-%d %H:%M:%S)";

[0235] S413. If the first parameter entered by the user in the console is not Imo or Name, an error message will be displayed to the user, and the error message content is "There is a problem with the order or name of the field identifier entered in Redis";

[0236] S414. If the number in the string input by the user in the console is inconsistent with the parameter passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0237] S415, if there is no any of the above errors, the four fields of the transmitted news data, including the news content, are converted into ASCII code and stored in the news update identification data;

[0238] S42. The method used when obtaining news user pictures for the first time is as follows:

[0239] Read_the_stored_News_data_and_identify method (parameters: website name, ship IMO number or name, news category, release time, news content):

[0240] S421. If the date format of the parameter passed is wrong, an error will be reported, and the error content is "Date storage error, please recheck and run this file";

[0241] S422, traverse the stored news update identification data, if it is a null value, report an error, the error content of which is "the execution file sequence or null data is wrong, please execute the file in sequence";

[0242] S423, judging whether the four parameter field data passed from the page have been stored in the news update identification data, and returning a record flag and True or False, wherein the news content data of the passed parameters are converted into ASCII code;

[0243] Read_the_stored_newsUsersPicturesUrl_data_and_identify method (parameters: website name, picture URL):

[0244] S424, traverse the stored news user picture Url data, determine whether the news user picture Url passed from the page has been stored, if not stored, store the news user picture Url in the news user picture Url data, and return a record flag, True or False;

[0245] S43. The method of using the news update file is as follows:

[0246] Determine_Whether_The_News_Needs_To_Update_And_Modify_Stored_Data method (parameters: website name, ship IMO number or name, news category, release time, news content):

[0247] S431. If the date format of the passed parameter is wrong, an error will be reported, and the error content is "Date storage error, please recheck and run this file";

[0248] S432, traverse the stored news update identification data, if it is a null value, then report an error, the error content of which is "execution file sequence or null data error, please execute the file in sequence";

[0249] S433. If the number in the string input by the user in the console is inconsistent with the parameter passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data."

[0250] S434. If there are no errors above, the console will prompt the user, and the prompt content is "Please enter the field content of the news update storage model, separated by spaces, and please strictly pass according to the method parameters. The first parameter is preferably the ship's Imo number as the identifier. If the Imo number is empty, the ship's Name can be passed as the identifier; if this website has no news category, the second parameter passes a null value. Please change the time parameter structure to something like (%Y-%m-%d %H:%M:%S)";

[0251] S435. When the user enters N / n in the console, an error message will be displayed, which reads "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful."

[0252] S436. If there are no errors in the above, the transmitted news content parameter value is converted into ASCII code, and the transmission is performed to determine whether the ship Imo or Name parameter field data value transmitted from the page is in the stored news update identification data. If it has not been stored, the news data is stored in the news update identification data and stored in the updated news update identification data; if it has been stored, determine whether the news category parameter field data value transmitted by the same ship is in the stored news update identification data. If it has not been stored, the news data is stored in the news update identification data and stored in the updated news update identification data; if it has been stored, under the conditions of the same ship and the same news category, determine the format of the transmitted news release time parameter field data value and convert it into a timestamp and compare it with the time extracted from the stored news update identification data. If the time parameter transmitted is If the time is newer than the stored news time, the news data is stored in the news update identification data and stored in the updated news update identification data; if the time is older, no processing is performed; if the time is equal, it is determined whether the ASCII code converted from the passed news content parameter value is the same as the ASCII code of the ship news stored in the news update identification data. If the news is the same, no processing is performed; if the news is not the same, the news data updated on that day is traversed and stored. If the news passed by the parameter is not in the news data updated on that day, all news of the same category of this ship are deleted, the update identification data of this ship news is updated, and the ASCII code converted from the passed news content parameter value is stored in the updated news data; if the ASCII code converted from the passed news content parameter value is in the updated news data, no processing is performed;

[0253] S437, returning a record flag indicating whether the news parameter value passed from the page needs to be updated, and returns True or False;

[0254] S44. The method for using the news user picture update file is as follows:

[0255] Read_the_stored_new_News_data_and_identify method (parameters: website name, ship IMO number or name, news category, release time, news content):

[0256] S441. If the date format of the passed parameter is wrong, an error will be reported, and the error content is "Date storage error, please recheck and run this file";

[0257] S442, traverse the updated news update identification data, if it is a null value, report an error, the error content of which is "execution file sequence or null data error, please execute the file in sequence";

[0258] S443, convert the passed news parameter value into ASCII code, determine whether the four parameter values ​​passed from the page are stored in the updated news update identification data, and return a record flag, True or False;

[0259] Read_the_stored_newsUsersPicturesUrl_data_and_identify method (parameters: website name, picture URL):

[0260] S444, traverse the news user picture Url data, determine whether the news user picture Url data passed from the page has been stored, and return a record flag, True or False;

[0261] S445: When the user obtains updated news data for the second time or multiple times, the updated news update identification data stored needs to be deleted and the ClearTheKeyForAddingAndUpdatingNewNews method is executed.

[0262] In this embodiment, some websites have news data themselves. Their characteristics are that one ship corresponds to different news, and the news on different websites is updated at different frequencies every day, and the news update frequency corresponding to each ship is also different. When comparing news content, the present invention adopts the method of converting news into ASCII code for storage. ASCII code is a character-based encoding method. After the news content is converted into ASCII code, the changes in the news content can be quickly compared by comparing the differences in ASCII codes. It is stored byte by byte by character, and the comparison process can be carried out more efficiently. Compared with directly comparing the original text content, comparing it can save computing resources and processing time. Its comparison is based on characters, avoiding complex text parsing and processing of text content, so it is more efficient when processing a large amount of news data. Since it is a character-based encoding method, it does not involve factors such as language and format, so the comparison process is more stable and reliable, reducing errors caused by language differences or format changes. ASCII code is a basic coding method that can be recognized by computers. It can realize automatic news content comparison and update detection through programming, reducing the need for human intervention and improving processing efficiency and accuracy. It is also a character encoding method widely supported in computer systems. It is suitable for comparison and processing needs on various platforms and environments and has good versatility and compatibility.

[0263] Most websites have news release times, and news corresponding to the same ship is updated based on the release time. However, it should be noted that the website may classify news, and the categories of news are independent. If the categories corresponding to the same ship are different, only the data of the same category needs to be updated. When the ship associated with the news cannot obtain the unique identification number Imo, the ship name can be used as the unique identification for storage to ensure the integrity of the data. The present invention stores the Vesselslmo (Vessels Name), Category, Uploader Date, and Article fields of the acquired news. One of the key points here is the news release time format. If the website's release time format is in the form of year, month, and day, if the news update frequency of the same ship on the same day is relatively high, repeated acquisition will cause the latest news data to be different from the actual storage time. The model method of the present invention opens up another space to store the news data that has been updated on the same day. In news about the same ship and type, but with different news contents, if the release time of the news obtained from the page is equal to the release time of the corresponding news stored for this ship, it is necessary to traverse and store the updated news data. If it has been stored, no processing will be done. If it has not been stored, it means that on the same day, the news about this ship is newer than the stored news and needs to be stored.

[0264] For the completeness of the model method, some websites’ news have information about the user who published the news. When the picture data of the news publisher is obtained for the first time, the picture’s URL data is saved, and the stored news data is traversed. If the news data has been stored and the picture’s URL data has not been stored, the picture of the news publisher is obtained. When the picture of the news publisher is updated, the stored updated news data is traversed. If the news data passed from the page has been stored in the updated news data and the picture’s URL data has not been stored, the picture of the news publisher is obtained. Ensure the consistency of the data stored in the two files.

[0265] Example

[0266] Experimental settings, parameters and method selection:

[0267] The experimental method adopted by the present invention is to obtain the website multimodal data verification model through Scrapy, test the data of different websites, and select parameters according to the characteristics of the specific website. Ensure that the Settings parameters of the Scrapy file used by the test model are certain.

[0268] Setting CONCURRENT_REQUESTS to a power of 2 is an optimization strategy, mainly to better utilize the computing system resources in Scrapy and improve the efficiency of acquisition. The principle of this setting is that in computers, the size of many underlying operations and data structures is a power of 2. Setting CONCURRENT_REQUESTS to a power of 2 can better match these underlying operations and data structures, thereby improving the efficiency of the system. In addition, some operating systems and network libraries will also optimize the number of concurrent requests by rounding to a power of 2, so setting CONCURRENT_REQUESTS to a power of 2 will also help adapt to these systems and libraries, reducing unnecessary resource waste and performance loss.

[0269] When all the parameters of Settings, storage method (setting Redis partitioning), and acquisition page are the same (in this embodiment, 5 pages of ship data are acquired), the optimal parameters of CONCURRENT_REQUESTS of this machine are tested, such as Figure 5 As shown in the figure, through test comparison, the optimal CONCURRENT_REQUEST value of this computer is 512. The larger the value of CONCURRENT_REQUESTS, the greater the server load, and may even trigger the server mechanism. When too many requests are sent at the same time, a lot of bandwidth resources may be occupied, especially in the case of slow network speed or narrow bandwidth, which will cause increased request delay or packet loss. Processing too many requests at the same time may increase memory consumption, especially when obtaining a large amount of data and generating a large amount of intermediate data.

[0270] By setting CONCURRENT REQUESTS to a fixed value of 512 and then setting DOWNLOAD DELAY to a smaller value, the acquisition speed can be controlled to avoid placing too much burden on the target website. Doing so can reduce and improve the sustainability of acquisition. Therefore, this embodiment uses CONCURRENT_REQUESTS=512, DOWNLOAD_DELAY=0.02, and the same 5 pages of ship data settings to test the acquisition time selected by different parameters and storage methods to find the optimal model method. Because the ship data of most websites is displayed in pages, each page can display a different number of ships, 9, 18, and 27 ships per page respectively. The experimental results of testing the number of items displayed per page are shown as follows. Figure 6As shown, the blue line is 9 ship data per page, the orange line is 18 ship data per page, and the green line is 27 ship data per page. When more data is displayed on one page, fewer network requests are required to obtain the complete data set. In contrast, displaying 9 data on one page may require more requests to obtain the complete data set, while displaying 27 data on one page can reduce the number of requests, thereby reducing the overall request time. In some cases, processing more data will result in better performance because system resources can be used more efficiently. The server returns different response times for different amounts of data. In some cases, the server optimizes requests for more data to reduce response time.

[0271] When all the parameters and acquisition pages of Settings are the same (this experiment uses 5 pages of ship data), the test model method uses array, Redis, and Redis block storage to obtain the time respectively, and selects the best method. When storing in Redis, all ship Imo are stored in the same key. When storing in Redis blocks, the ships are classified by obtaining the first letters of the ship names and stored in different keys. The above three methods are tested by the time used for ship structured update files. The results are as follows Figure 7 As shown, the blue lines are for storing data in arrays, the orange lines are for storing data in Redis (single key), and the green lines are for storing data in Redis (multi-key).

[0272] An array (or list) is usually a continuous space in memory. When the amount of data is large, the time complexity of read and write operations is O(1), but as the amount of data increases, array insertion and deletion operations may become slower because a large number of elements need to be moved. Redis uses a memory-based key-value storage system, and data is stored in key-value pairs in memory rather than in arrays. Because Redis uses efficient data structures such as hash tables and implements fast read and write operations, it can provide smaller read and write time consumption.

[0273] In Redis single-key mode, all data is stored in one key. When the amount of data is large, the read and write operations of a single key may become a bottleneck, especially in a high-concurrency environment. In multi-key (block) mode, data is divided into multiple keys, which allows more parallel processing. This means that multiple small blocks of data can be read and written at the same time, thereby improving read and write efficiency and overall throughput. Single-key operations can cause performance bottlenecks due to competition, especially in a high-concurrency environment. The multi-key (block) method can disperse and store data in multiple keys, thereby reducing the read and write pressure of a single key, reducing competition, and improving the concurrency of the system. In single-key mode, the reading and writing of large amounts of data may cause network transmission to become a bottleneck, especially when the amount of data is large. In multi-key (block) mode, network utilization can be improved by parallel transmission between multiple keys, thereby reducing the time consumption of data transmission.

[0274] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for acquiring and updating maritime multimodal data, characterized in that: include: Obtain maritime multimodal ship data, port data, and news data; According to the acquired maritime multimodal ship data, the multimodal ship data is updated based on the ship's first letter; Update multimodal port data based on the acquired port data; Update the multimodal news data based on the acquired news data.

2. A method for acquiring and updating maritime multimodal data according to claim 1, characterized in that: The acquisition of maritime multimodal ship data, port data, and news data specifically includes: Obtain maritime multimodal ship data. Basic ship information is structured data, which exists in the form of tables or database records, which is easy to classify and analyze; ship images and ship trajectories are unstructured data, which are processed and analyzed through specific technologies and tools; Obtain port data. The basic information, ship arrival time, departure time, and cargo loading and unloading status in the port data are structured data, while the port image data is unstructured data. Get news data. The publisher and publishing time in the news data are structured data, while the news content is unstructured data. Get the corresponding data dictionary, which exists in the website structured data; Observe the data and check whether the website requires login. Since the number of ships displayed on each page of some websites can be adjusted, find the optimal parameters for displaying each page of data according to the time of obtaining data to improve the acquisition speed. After successfully obtaining the first page of data, calculate the number of pages of data that need to be obtained, obtain the total number through the first request, and then calculate how many pages of data need to be obtained based on the number of each page; The field data related to the ship structured data text update time and the ship construction time are expressed in the form of timestamps. The timestamps are calculated from 00:00:00 on January 1, 1970. The timestamps before this time point are negative numbers, and the timestamps after this time point are positive numbers.

3. The method for acquiring and updating maritime multimodal data according to claim 1, characterized in that: The updating of the multimodal ship data based on the first letter of the ship according to the acquired maritime multimodal ship data specifically includes: The method used to obtain ship structured data for the first time is as follows: Traverse the stored ship's newly added identification data, determine whether the ship's Imo number data passed by the parameter exists in the ship's newly added identification data, if not, store the ship's Imo number data passed by the parameter in the ship's newly added identification data, and finally return True or False; When you need to store any field representing the ship update, the console will prompt the user to enter the field content of the ship update storage model, separated by spaces, and strictly follow the item class sub-segment naming. Please be sure to store Imo as the first parameter. If there is a time parameter, please change the time structure to something similar; If the string entered by the user in the console is not recognized as Imo or imo, an error message will be displayed to the user, indicating that there is a problem with the order or naming of the field identifiers entered in Redis; If the number of characters in the string entered by the user in the console is inconsistent with the parameters passed by the method, all stored data will be deleted and an error will be reported. The error content is that the field length provided by the console is inconsistent with the length of the passed parameter; the console provides the specific data of the field length and the specific data of the length of the passed parameter, and displays the content "Sorry! Please re-store all data"; If there is no error in any of the above, the update identification field corresponding to the ship's Imo number is stored in the ship update identification data; The method used to obtain ship image data for the first time is as follows: Traverse the stored ship's newly added identification data. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence"; If there are no errors above, it will determine whether the Imo number passed from this page has been stored, and return True or False; The method used by the newly added file of ship structured data is as follows: Traverse the stored new ship identification data to determine whether the Imo number passed from this page has been stored. If it has not been stored, store the first letter of the newly added ship, store the Imo number of this ship in the new ship identification data or store it in the new identification data of the newly added ship, and store the first letter of the ship in the new ship first letter data; Use the Imo number passed from the page to determine whether it has been stored before as a mark and return True or False; When a new ship appears, the console will prompt the user, prompting "Please enter the length and order of the field content of the ship storage model in the new file and the field content stored in Redis. Please be sure to store Imo as the first parameter. If there is a time parameter, please change the time structure to be similar"; When the user enters other letters, the system will continue to allow the user to enter; When the user enters N / n, all stored data will be deleted and an error will be reported, with the error content being "Sorry! Please re-store all data"; When the number of field parameters passed is inconsistent with the number of storage memory fields, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data." Traverse the stored ship's newly added identification data. If there is no data, an error will be reported. The error content is "Error in the execution file sequence or empty data. Please execute the file in sequence"; If there is no error in any of the above, the update identification field corresponding to the ship's Imo number will be stored in the ship's update identification data; The method used to obtain the newly added ship image data is as follows: Traverse the stored data of the newly added ship's first letters. If there is data, return all the data; if there is no data, report an error with the error content "There is currently no new ship's first letter information"; Traverse and store the newly added identification data of the newly added ship. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence"; Determines whether the ship IMO number passed from the page is in the newly added identification data of the newly added ship, and will return True or False; The method used to update the ship structured data file is as follows: Traverse the stored ship update identification data. If it is a null value, an error is reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence"; When the number of field parameters passed is inconsistent with the number of storage memory fields, an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Please modify it and re-run this file." When a ship with updated data appears, the console will prompt the user, prompting "Please enter the length and order of the field content of the ship storage model in the update file and the field content stored in Redis. Please be sure to store Imo as the first parameter. If there is a time parameter, please change the time structure to be similar"; When the user enters N / n in the console, an error message will be displayed, which reads "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful." If there is no error in any of the above, based on the same ship Imo number, compare the ship update identification field value passed from the page with the stored update identification field value to see if they are the same. If they are different, update the update identification field data corresponding to the stored ship Imo number; Returns the record flag of whether the ship data passed from the page needs to be updated, returning True or False; When the user obtains the newly added ship data for the second time or multiple times, the newly added identification data of the newly added ship needs to be deleted.

4. The method for acquiring and updating maritime multimodal data according to claim 1, characterized in that: The updating of the multimodal port data according to the acquired port data specifically includes: The method used to obtain port structured data for the first time is as follows: Traverse the stored port new identification data, determine whether the port UN_Locode, Name, Country data passed by the page has been stored, if not, store the passed port UN_Locode, Name, Country data in the port new identification data, and finally return True or False; When a port needs to be stored in the port update identification data, the console will prompt the user, and the prompt content is "Please enter the field content of the port update storage model, separated by spaces, please strictly follow the item class field naming, and be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter)"; If the user enters parameters in a different order in the console, an error message will be displayed, saying "There is a problem with the order or naming of the field identifiers entered in Redis"; If the number of characters in the string entered by the user in the console is inconsistent with the parameters passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data." If there is no error in any of the above, the update identification field corresponding to the port is stored in the update identification data; The method used to obtain port image data for the first time is as follows: Traverse the stored port new identification data. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence"; If there is no error, determine whether the UN_Locode, Name, Country data passed by the page has been stored, and return True or False; The method used by the newly added file of port structured data is as follows: Traverse the newly added identification data of the port to determine whether the port UN_Locode, Name, Country data passed by this page has been stored. If not, store the port UN_Locode, Name, Country data passed by the page in the newly added identification data of the port or in the newly added identification data of the newly added port; Returns whether the port UN_Locode, Name, Country data passed from the page has been stored as a mark, and returns True or False; When a new port appears, the console will prompt the user, prompting "Please enter the length and order of the field content of the port storage model in the new file to ensure that it is consistent with the field content stored in Redis. Please be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter (Y / N)"; When the user enters other letters, the system will continue to allow the user to enter; When the user enters N / n, all stored data will be deleted and an error will be reported, with the error content being "Sorry! Please re-store all data"; When the number of field parameters passed is inconsistent with the number of storage memory fields, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data." Traverse the port to update the identification data. If there is no data, an error will be reported. The error content is "Error in the execution file sequence or empty data, please execute the file in sequence"; If there is no one of the above errors, the port update identification field passed by the parameter is stored in the update identification data; Returns whether the port update identification field passed from the page is a newly added port record flag, and returns True or False; The method used to obtain the newly added port image data is as follows: Traverse the newly added identification data of the newly added port. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence"; Determines whether the port UN_Locode, Name, Country data passed from the page is in the newly added identification data of the newly added port, and returns True or False; The method used to update the port structured data file is as follows: Traverse the port to update the identification data. If it is a null value, an error will be reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence"; When the number of field parameters passed is inconsistent with the number of storage memory fields, an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Please modify it and re-run this file." When a port needs to update data, the console will prompt the user, prompting "Please enter the length and order of the field content of the port storage model in the newly added file and the field content stored in Redis. Please be sure to store UN_Locode as the first parameter, Port_Name as the second parameter, and Port_Country as the third parameter (Y / N)"; When the user enters N / n in the console, an error message will be displayed, which reads "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful." If there is no error in any of the above, based on the same port UN_Locode, Name, Country data, compare the port update identification field value passed from the page with the stored update identification field value to see if they are the same. If they are different, update the corresponding update identification field data of the stored port; Returns whether the port data passed from the page is the record flag of the updated port, and returns True or False; When the user obtains the newly added port data for the second time or multiple times, it is necessary to delete the stored newly added identification data of the newly added port.

5. The method for acquiring and updating maritime multimodal data according to claim 1, characterized in that: The updating of the multimodal news data according to the acquired news data specifically includes: The method used to obtain news for the first time is as follows: If the date format of the passed parameter is incorrect, an error will be reported, and the error content is "Date storage error, please recheck and run this file"; When there is news content that needs to be stored, the console will prompt the user, and the prompt content is "Please enter the field content of the news update storage model, separated by spaces, and pass it strictly according to the method parameters. The first parameter is preferably the ship's Imo number as the identifier. If the Imo number is empty, the ship's Name can be passed as the identifier; if this website has no news category, the second parameter passes an empty value. Please change the time parameter structure to something similar"; If the first parameter entered by the user in the console is not Imo or Name, an error message will be displayed to the user, with the error content being "There is a problem with the order or name of the field identifiers entered in Redis"; If the number of characters in the string entered by the user in the console is inconsistent with the parameters passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data." If there is no error in any of the above, the four fields of the transmitted news data, including the news content, are converted into ASCII code and stored in the news update identification data; The method used to obtain news user pictures for the first time is as follows: If the date format of the passed parameter is incorrect, an error will be reported, and the error content is "Date storage error, please recheck and run this file"; Traverse the stored news update identification data. If it is a null value, an error is reported. The error content is "Error in the execution file sequence or null data. Please execute the file in sequence"; Determine whether the four parameter field data passed from the page have been stored in the news update identification data, and return the record flag and True or False, wherein the news content data of the passed parameters is converted into ASCII code; Traverse the stored news user picture Url data, determine whether the news user picture Url passed from the page has been stored, if not, store the news user picture Url in the news user picture Url data, and return the record flag, True or False; The method used by the news update file is as follows: If the date format of the passed parameter is incorrect, an error will be reported, and the error content is "Date storage error, please recheck and run this file"; Traverse the stored news update identification data. If it is a null value, an error is reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence"; If the number of characters in the string entered by the user in the console is inconsistent with the parameters passed by the method, all stored data will be deleted and an error will be reported. The error content is "The field length provided by the console is inconsistent with the length of the passed parameter. The console provides field length information and the length information of the passed parameter. Sorry! Please re-store all data." If there are no errors above, the console will prompt the user, and the prompt content is "Please enter the field content of the news update storage model, separated by spaces, and pass it strictly according to the method parameters. The first parameter is preferably the ship's Imo number as the identifier. If the Imo number is empty, the ship's Name can be passed as the identifier; if this website has no news category, the second parameter passes an empty value. Please change the time parameter structure to something similar"; When the user enters N / n in the console, an error message will be displayed, which reads "You think the fields are different. Please compare the fields in the file and re-run this update file after the modification is successful." If there are no errors in the above, the passed news content parameter value is converted into Ascii code, and the passed data is judged whether the ship Imo or Name parameter field data value passed from the page is in the stored news update identification data. If it has not been stored, the news data is stored in the news update identification data and stored in the updated news update identification data; if it has been stored, it is judged whether the news category parameter field data value passed by the same ship is in the stored news update identification data. If it has not been stored, the news data is stored in the news update identification data and stored in the updated news update identification data; if it has been stored, under the conditions of the same ship and the same news category, the format of the passed news release time parameter field data value is judged and converted into a timestamp and compared with the time extracted from the stored news update identification data. If the time passed by the parameter is longer than the stored time, then the news is saved in the news update identification data. The stored news time is updated, the news data is stored in the news update identification data, and stored in the updated news update identification data; if the time is older, no processing is performed; if the time is equal, it is determined whether the ASCII code converted from the passed news content parameter value is the same as the ASCII code of the ship news stored in the news update identification data. If the news is the same, no processing is performed; if the news is not the same, the news data updated on that day is traversed and stored. If the news passed by the parameter is not in the news data updated on that day, all news of the same category of this ship are deleted, the update identification data of this ship news is updated, and the ASCII code converted from the passed news content parameter value is stored in the updated news data; if the ASCII code converted from the passed news content parameter value is in the updated news data, no processing is performed; Returns the record flag of whether the news parameter value passed from the page needs to be updated, and returns True or False; The method used for news user picture update files is as follows: If the date format of the passed parameter is incorrect, an error will be reported, and the error content is "Date storage error, please recheck and run this file"; Traverse the updated news update identification data. If it is a null value, an error is reported. The error content is "The execution file sequence or null data is wrong. Please execute the file in sequence"; Convert the passed news parameter value into ASCII code, determine whether the four parameter values ​​passed from the page are stored in the updated news update identification data, and return the record flag, True or False; Traverse the news user picture Url data, determine whether the news user picture Url data passed from the page has been stored, and return the record flag, True or False; When the user obtains updated news data for the second time or multiple times, it is necessary to delete the stored updated news update identification data.

Citation Information

Cited By

  • Intelligent seaport-oriented AI knowledge base construction method and system

    CN120973959A